# O'FAIRe makes you an offer: Metadata-based Automatic FAIRness Assessment for Ontologies and Semantic Resources

Emna Amdouni, Syphax Bouazzouni, Clement Jonquet

## ► To cite this version:

Emna Amdouni, Syphax Bouazzouni, Clement Jonquet. O'FAIRe makes you an offer: Metadata-based Automatic FAIRness Assessment for Ontologies and Semantic Resources. International Journal of Metadata, Semantics and Ontologies, 2022, 16 (1), pp.16-46. 10.1504/IJMSO.2022.131133. lirmm-03630233v2

HAL Id: lirmm-03630233

https://hal-lirmm.ccsd.cnrs.fr/lirmm-03630233v2

Submitted on 30 May 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# O'FAIRe makes you an offer: metadata-based automatic FAIRness assessment for ontologies and semantic resources

Emna Amdouni and
Syphax Bouazzouni

LIRMM,
University of Montpellier & CNRS,
Montpellier, France
Email: emna.amdouni@lirmm.fr
Email: syphax.bouazzouni@lirmm.fr

Clement Jonquet*

LIRMM,
University of Montpellier & CNRS,
Montpellier, France
and
MISTEA,
University of Montpellier, INRAE & Institut Agro,
Montpellier, France
Email: jonquet@lirmm.fr
*Corresponding author

**Abstract:** We have not yet seen a clear methodology implemented and tooled to automatically assess the level of FAIRness of semantic resources. We propose a metadata-based automatic FAIRness assessment methodology for ontologies and semantic resources called Ontology FAIRness Evaluator (O'FAIRe). It is based on the projection of the 15 foundational FAIR principles for ontologies, and it is aligned and nourished with relevant state-of-the-art initiatives for FAIRness assessment. We propose 61 questions of which 80% are based on the resource metadata descriptions and we review the standard metadata properties (taken from the MOD 1.4 ontology metadata model) that could be used to implement these metadata. We also demonstrate the importance of relying on ontology libraries or repositories to harmonise and harness unified metadata and thus allow FAIRness assessment. Moreover, we have implemented O'FAIRe in the AgroPortal semantic resource repository and produced a preliminary FAIRness analysis over 149 semantic resources in the agri-food/environment domain.

**Keywords:** automatic FAIR assessment; semantic web; ontologies; standardised metadata and open repositories.

**Reference** to this paper should be made as follows: Amdouni, E., Bouazzouni, S. and Jonquet, C. (2022) ‘O’FAIRe makes you an offer: metadata-based automatic FAIRness assessment for ontologies and semantic resources’, *Int. J. Metadata Semantics and Ontologies*, Vol. 16, No. 1, pp.16–46.

**Biographical notes:** Emna Amdouni, PhD in Informatics, is a senior engineer in semantic web and knowledge technologies. She is a former postdoc at LIRMM laboratory CNRS & University of Montpellier (2 years) and a former research assistant at the University of Lyon 2 (2 years). She has eight years of experience in ontologies and semantic web research applied to biomedicine, industry, and, more recently, open science. In particular, she works on formalising semantic requirements, implementing knowledge solutions (RDF, OWL, JSON-LD, SPARQL, SWRL) to industrial partners, and FAIRification of semantic data and metadata to improve their quality. She is one of the principal contributors to the IndustryPortal (https://industryportal.enit.fr/), a public portal built to serve the semantic industrial community, funded by the H2020 OntoCommons project (https://ontocommons.eu/).

Syphax Bouazzouni is a computer science engineer in Information Systems and Technology. He is an expert in web development and proficient with semantic web technologies and ontology-based services. He is the lead developer for the AgroPortal project (https://agroportal.lirmm.fr/)

# O'FAIRe makes you an offer: metadata-based automatic FAIRness assessment for ontologies and semantic resources

Emna Amdouni and
Syphax Bouazzouni

LIRMM,
University of Montpellier & CNRS,
Montpellier, France
Email: emna.amdouni@lirmm.fr
Email: syphax.bouazzouni@lirmm.fr

Clement Jonquet*

LIRMM,
University of Montpellier & CNRS,
Montpellier, France
and
MISTEA,
University of Montpellier, INRAE & Institut Agro,
Montpellier, France
Email: jonquet@lirmm.fr
*Corresponding author

**Abstract:** We have not yet seen a clear methodology implemented and tooled to automatically assess the level of FAIRness of semantic resources. We propose a metadata-based automatic FAIRness assessment methodology for ontologies and semantic resources called Ontology FAIRness Evaluator (O'FAIRe). It is based on the projection of the 15 foundational FAIR principles for ontologies, and it is aligned and nourished with relevant state-of-the-art initiatives for FAIRness assessment. We propose 61 questions of which 80% are based on the resource metadata descriptions and we review the standard metadata properties (taken from the MOD 1.4 ontology metadata model) that could be used to implement these metadata. We also demonstrate the importance of relying on ontology libraries or repositories to harmonise and harness unified metadata and thus allow FAIRness assessment. Moreover, we have implemented O'FAIRe in the AgroPortal semantic resource repository and produced a preliminary FAIRness analysis over 149 semantic resources in the agri-food/environment domain.

**Keywords:** automatic FAIR assessment; semantic web; ontologies; standardised metadata and open repositories.

**Reference** to this paper should be made as follows: Amdouni, E., Bouazzouni, S. and Jonquet, C. (2022) ‘O’FAIRe makes you an offer: metadata-based automatic FAIRness assessment for ontologies and semantic resources’, *Int. J. Metadata Semantics and Ontologies*, Vol. 16, No. 1, pp.16–46.

**Biographical notes:** Emna Amdouni, PhD in Informatics, is a senior engineer in semantic web and knowledge technologies. She is a former postdoc at LIRMM laboratory CNRS & University of Montpellier (2 years) and a former research assistant at the University of Lyon 2 (2 years). She has eight years of experience in ontologies and semantic web research applied to biomedicine, industry, and, more recently, open science. In particular, she works on formalising semantic requirements, implementing knowledge solutions (RDF, OWL, JSON-LD, SPARQL, SWRL) to industrial partners, and FAIRification of semantic data and metadata to improve their quality. She is one of the principal contributors to the IndustryPortal (https://industryportal.enit.fr/), a public portal built to serve the semantic industrial community, funded by the H2020 OntoCommons project (https://ontocommons.eu/).

Syphax Bouazzouni is a computer science engineer in Information Systems and Technology. He is an expert in web development and proficient with semantic web technologies and ontology-based services. He is the lead developer for the AgroPortal project (https://agroportal.lirmm.fr/)
and a major contributor to the OntoPortal project (https://github.com/ontoportal) which generalises the technology for ontology repositories. His main activities are to develop and maintain the AgroPortal platform, collaborate and help on lead the technical management changes in the OntoPortal Alliance.

Clement Jonquet, PhD in Informatics, is senior researcher at INRAE and associate researcher at University of Montpellier. He is a former postdoc (3 years) & visiting scholar (3 years) from Stanford BMIR. He has 15 years of experience in ontologies and semantic Web research applied to biomedicine and agronomy. He works on the design and development of ontology repositories, ontology-based services and FAIR semantic artefacts as the PI or Co-PI of projects such as ANR D2KAB or Horizon-Europe FAIR-IMPACT. Since 2015, he leads the AgroPortal initiative. Dr. Jonquet is the (co)author of 129 publications or communications (+3500 citations).

# 1 Introduction

Making digital scientific data openly available remains a big challenge for both the scientific community and funding agencies. In this context, in 2014, a group of researchers, research institutions and publishers – called FORCE 11 – committed to making scientific data interoperable, persistent and understandable for both humans and machines. They defined fundamental guiding principles called FAIR (for Findable, Accessible, Interoperable and Reusable) that aim to conduct data stewards and publishers in their implementation choices (Wilkinson et al., 2016). Since then, the ‘FAIR movement’ has been broadly embraced by scientists, data stakeholders, agencies and governance bodies. Mons’s (2018) book defines the FAIR principles as not a standard nor a technology but are only specifications for digital objects. Their different aspects are related but need to be assessed independently.

In open science, the expression ‘FAIRness assessment’ (or ‘FAIRness compliance’) refers to evaluating to which degree a digital object adheres to the FAIR principles. It is complex as the FAIR principles are expressed at a very generic level and need to be expanded and projected to specific data objects to be more explicit. Despite the wide embracement of the FAIR principles, no universal FAIRness assessment criteria exist. Furthermore, some criteria are tricky to evaluate automatically and require – sometimes subjective – human expertise, e.g., *meeting domain-relevant community standards* (R1.3) or *detailed provenance* (R1.2). Therefore, to enable FAIRness assessment and cover all the FAIR principles, it is preferable to distinguish what relies on human decision (e.g., which license to assign, which good practices to follow) from how to capture or represent information in a way a machine can use to evaluate FAIRness. We have adopted this approach, as explained after. The development of ‘metrics’ – i.e., a way to measure to which level a digital object implements FAIR principles – remains challenging. Several schemas and tools are proposed to manually or automatically assess the FAIRness of data sets, i.e.: FAIR Metrics (Wilkinson et al., 2018, 2019), the FAIR Data Maturity Model (FDMM) (Bahim et al., 2020), FAIRDat, FAIR-Aware (Devaraju et al., 2021) and more detailed later. However, we believe only the FAIR Metrics group proposes

a framework with some universal metrics for automatising the evaluation of FAIR digital objects. Most other approaches merely provide FAIR scoring based on forms or self-assessment questionnaires and recommendations on how to improve FAIR scores.

Semantic technologies are essential in making data interoperable and reusable, as required by ‘I’ and ‘R’ principles. In fact, standard vocabularies or ontologies are key to achieving a high level of FAIRness as stated by the ‘I2’ sub-principle: (Meta)data use vocabularies that follow FAIR principles. However, like any other data, semantic resources and ontologies¹ – the backbone of semantic technologies – have themselves to be FAIR. Until recently, not much attention has been made to describing and evaluating ontologies using FAIR principles. Early 2018, we argued that rich metadata descriptions and ontology repositories offer a means to facilitate the implementation of ‘FAIR ontologies’ (Jonquet, 2018). Later, we demonstrated the impact of harmonised and standardised metadata descriptions on the ontology identification and selection process (Jonquet et al., 2018a). This work was accomplished in part in the context of the Research Data Alliance VSSIG² and the AgroPortal project, a vocabulary and ontology repository dedicated to agri-food (Jonquet et al., 2018b). More recently, other community efforts have also expressed the need for recommendations and guidelines on how to provide FAIR semantic resources or ‘artefacts’ including the FAIRsFAIR H2020 project (Le Franc et al., 2020), or expert group guidelines (Garijo and Poveda-Villalón, 2020; Malone et al., 2016). However, these works focus on recommendations and guidelines but do not specify a methodology for assessing the FAIRness of semantic resources and automating this task.

In a previous paper, we introduced an *integrated quantitative FAIRness assessment grid* for semantic resources (Amdouni and Jonquet, 2021). This grid dispatches different credits to each FAIR principle, depending on its importance when assessing the FAIRness of semantic resources. In this paper, we go one step beyond and provide a clear methodology, implemented and tooled to automatically assess the level of FAIRness to guide semantic stakeholders to make their semantic resources FAIR and select relevant FAIR semantic resources for use. This methodology considers FAIRness assessment of ontologies should as much as
possible be based on the evaluation of their metadata
properties, which ones shall be ideally indexed, shared and
standardised by reference ontology repositories or libraries
(Ding and Fensel, 2001; D'Aquin and Noy, 2012). We
believe that clear metadata descriptions and open semantic
repositories are two mandatory elements to make semantic
resources FAIR. In previous work, we have built a new
harmonised metadata model for an ontology repository and
demonstrated that it enhances the identification and reuse of
ontologies (Jonquet et al., 2018a). This paper demonstrates
that ontology repositories are also crucial for FAIRness
assessment.

We build an automatic FAIRness assessment methodology for semantic resources. This methodology uses as much as possible assigned metadata values to answer a series of questions, specialised for ontologies and semantic resources, for each FAIR principle.³ We define 61 machine-actionable questions that describe unambiguously the tests to assess to which level a semantic resource respects a certain aspect of FAIR. This list of questions can be adapted or completed to refine the evaluation of one specific aspect without changing the overall methodology and assessment grid. We illustrate with some examples how the questions can be answered with a metadata property or an ontology repository feature. The methodology proposed is fully generic (i.e., does not depend on the ontologies, the domain or any specific technology) but requires ontologies to be described with an extensive set of metadata properties that go beyond what ontology developers and most repositories usually provide (Toulet et al., 2018).

We implemented this methodology in a FAIRness
assessment tool – called *Ontology FAIRness Evaluator*
(O'FAIRe) – which executes the tests automatically,
evaluating how a semantic resource responds to the questions.
O'FAIRe provides a global normalised FAIR score, i.e., a
number between 0 and 100 that can be detailed for each FAIR
principle (F, A, I, R) and sub-principle (F1,... R1.3). In
addition, the tool provides a short explanation for scores.
O'FAIRe is offered to: (i) enhance the level of FAIRness of a
semantic resource; (ii) help end users select FAIR ontologies
and thus respect the I2 principle. O'FAIRe's assessment
methodology is aligned with relevant state-of-the-art
initiatives for FAIRness assessment of semantic resources or
digital objects in general, and we have conceived it in a way
that can be customised, extended or improved by other
semantic experts in further studies. We considered the
FDMM, SHARC, 5-stars, FAIRsFAIR and Villalón-Poveda
et al. (2020) initiatives (as detailed in Section 2) and blend
them to determine a maximum number of credits for
assessing each FAIR sub-principle for a semantic resource.

We implemented O'FAIRe as a web service working with
any OntoPortal installations respecting the Metadata for
Ontology Description and Publication Ontology (MOD1.4)
metadata profile (Dutta et al., 2017). We deployed it in
AgroPortal (http://agroportal.lirmm.fr) and designed specific
user interfaces to experience its performance on ontologies or
groups of ontologies. Those preliminary results allowed us to

test and validate the accuracy of our tool and to develop
appropriate visualisations to display results graphically. This
paper also presents a synthesised analysis of FAIRness over
149 semantic resources in the agri-food/environment domain
available in AgroPortal as of January 2022. We mainly
observed that Findability and Accessibility were better
addressed than Interoperability and Reusability.

The rest of this paper is organised as follows: Section 2
introduces the FAIR principles and reviews FAIRness
assessment-related work (i.e., generic and specific ones.
Section 3 describes steps to design our metadata-based
FAIRness assessment methodology, details our approach by
presenting a projection of each FAIR principle for semantic
resources, lists the questions with some illustration examples,
and the metadata properties. Section 4 presents the O'FAIRe
prototype implemented in AgroPortal and then provides a
synthesised analysis for 149 semantic resources. Finally,
Sections 5 and 6 discuss some lessons learned from our
analysis and mention some future works before concluding
the paper.

## 2 Background on FAIRness assessment

In this section, we briefly present the FAIR principles and
influential actors in the ‘FAIR movement.’ Then, we present
FAIRness approaches in the literature, distinguishing two
categories: (i) generic recommendations, schemes or tools for
any kind of digital object (or for data set); (ii) approaches
specific for the description and assessment of ontologies or
semantic resources. We review both of them chronologically.

2.1 Terminology used

We use the following terminology, in part aligned with the
recent-and-being-developed *FAIR Vocabulary*,⁴ for referring
to ‘main’ FAIR related concepts:

* We use the term ‘*FAIR principle*’ to refer to each F, A, I, R guidelines group.

* We call ‘*FAIR sub-principle*’ the lower granularity for each FAIR principle (e.g., F1, F2, F3 and F4).

* ‘*FAIRness assessment*’ refers to the use of questions/tests to assess to which degree a semantic resource is compliant with a sub-principle or, by generalisation, a principle.

* A ‘*FAIRness assessment question*’ assign a certain number (maximised) of ‘*credits*’ to an ontology depending on how it passes the test. When assigned, credits become ‘*points*.’ The higher the number of points, the better the test is passed.

* ‘*FAIRness level*’ refers to how much a FAIR principle or a sub-principle is respected. this level will be represented by a ‘*FAIR score*’ i.e., the total number of points obtained by a semantic resource for a sub-principle, principle or overall. A score can be normalised between 0 and 100 for a more straightforward representation and comparison.
## 2.2 Overview of the 'FAIR movement'

Hereafter, we briefly introduce the meaning of each F-A-I-R aspect: First, data are *findable* when they are sufficiently described with metadata and are registered or indexed in a searchable registry or repository. Data, metadata and other associated resources should have a unique and persistent identifier that makes them findable and referenceable by humans and machines. Second, data are *accessible* when users can retrieve them using a universally implementable and open protocol. But this does not mean data have to be openly accessible without restrictions. Sometimes, data can be FAIR and not open. In other words, FAIR data should be associated with metadata that specifies conditions by which the data are accessible. Third, data are *interoperable* when other stakeholders can easily and in a standardised way process them without the need for specific software. The 'I' principles might be considered the more challenging to accomplish and still the most important key features for FAIR. Data and metadata should be described in a formal, accessible, shared and broadly applicable language for knowledge representation. Plus, data must themselves reuse FAIR vocabularies or ontologies and include qualified references to other data and metadata. Early on, semantic web and linked data technologies were identified as some best candidates to use for knowledge representation, machine-readability and interoperability on the web, but the FAIR principles cannot be reduced to the Semantic Web (Mons et al., 2017) and are voluntarily detached from any technology stack. Finally, data are *reusable* when they are provided with explicit license and data usage information for humans and machines. They should also be associated with rich metadata and documentation that detail their provenance (data specifications, funding projects, use cases, versions, experimental processes, etc.).

The FAIR principles are only described at a foundational level and have not been well defined at the implementation level. In Europe, e.g., the implementation of FAIR principles is supported, in part, by the *European Open Science Cloud* program of the European Commission, specifically by the expert group on FAIR data (Collin et al., 2018). The GO FAIR initiative has an objective to push for the adoption of the FAIR principles, especially via the description and adoption of *FAIR Implementation Profiles* and deployments of *FAIR Data Points*. Among other international initiatives pushing for the adoption of the FAIR principles, we can cite: Australia's *F.A.I.R. policy statement*,⁵ US's NIH Data Commons program,⁶ the Research Data Alliance (Bahim et al., 2020;

David et al., 2020), multiple H2020 projects including FAIRsFAIR,⁷ community-specific initiatives such as the *Food System* or other GO FAIR Implementation Network.⁸

Over the last couple of years, several working groups have formed to establish specifications and norms for FAIR. For example, the *FAIR digital object* model is a specification for implementing FAIR (Schwardmann, 2020). A FAIR digital object is composed of four elements, namely, the digital object itself, identifiers, standards and metadata. It ensures the interoperability of digital data via the use of a persistent identifier associating each data with its contextual metadata and a standard format to facilitate its reuse. Furthermore, digital objects could be set in a *FAIR ecosystem* (Collins et al., 2018) proposing services or infrastructure to use data as, e.g., the FAIR Digital Framework proposed in 2020. Table 1 shows the different elements of a FAIR ecosystem for semantic resources.

## 2.3 Generic FAIRness assessment approaches

The Research Data Alliance (RDA) *SHARing Rewards and Credit* (SHARC) Interest Group, David et al. (2017) proposed a FAIRness assessment grid to enable researchers and other data stakeholders to evaluate FAIR implementations and provide the appropriate means for crediting and rewarding to facilitate data sharing. The SHARC grid defines a set of 45 generic criteria with importance levels (essential, recommended, or desirable) evaluated by answering one of four values (never/NA, if mandatory, sometimes, always) to a question; questions are sometimes dependent on one another as in a decision tree. In 2018, the RDA *FAIR Data Maturity Model* (FDMM) Working Group published a recommendation to normalise FAIRness assessment approaches and enable comparison of their results (Bahim et al., 2020). It describes a set of 47 generic criteria derived from the FAIR principles with priorities (essential, important or useful). Both the SHARC grid and the FDMM recommendations assumed that some FAIR principles were more important than others. We have kept this philosophy in our methodology and kept the SHARC and FDMM outputs to influence our FAIRness assessment grid.

Some FAIRness assessment tools recently appeared, including FAIRdat, FAIR metrics, OzNome 5-star tool, FAIR self-assessment, FAIR-Aware, F-UJI, FAIR cookbook. We describe some of them hereafter and illustrate several graphical visualisations of FAIRness outputs in Figure.

<table><caption>**Table 1** FAIR ecosystem layers (Collins et al., 2018) and corresponding elements for semantic resources</caption><thead><tr><td>Applications and tools proposing content-based services for stakeholders</td><td>FAIR digital objects representing data in standard formats, with identifiers and metadata</td><td>Registries and repositories suggesting discovery, accessing, storage, and preservation services (ideally certified, trustworthy)</td><td>Networking infrastructure ensuring storage and processing of data, services, and automated workflows</td></tr></thead><tbody><tr><td>Knowledge-based systems or applications relying on ontologies either directly or thru repositories, e.g., CEDAR workbench, PHIS, COPO, etc.</td><td>Semantic resource encapsulated in a file typically with a knowledge representation format (OWL, SKOS, OBO, etc.) and syntax (RDF/XML, TTL, JSON-LD, etc.). Describe a URI and metadata properties (dc: creator, cc: license, etc.)</td><td>Ontology libraries (metadata only) such as OBO Foundry, FAIRsharing, BARTOC and repositories (metadata and content services) such as BioPortal, OLS, Ontobee, AgroPortal, EcoPortal, LOV, etc.</td><td>W3C compliant specifications for the web from HTTP and URIs to RDF, SPARQL or SHACL</td></tr></tbody></table>
Data Archiving and Networked Services (DANS) developed in 2017 *FAIRdat*,⁹ it is addressed to data reviewers/curators; it creates a badge scheme per principle called ‘FAIR profile’ that evaluates on a 5-stars scale how much a data set is compliant to each FAIR principle. For example, an evaluation result as ‘F4-A3-I2-R3’ denotes that the data set is easily findable, accessible under some conditions, has a low degree of interoperability, and is, on average reusable. This tool has been influenced by the Open Data Certificate¹⁰ and Berners-Lee’s 5-star scheme.¹¹ Considering the feedback of Fairdat’s participants in the testing phase of the first pilot version, at the Open Science Fair in 2017, the questionnaire is oversimplified, some criteria are missing, and the 5-stars rating system is subjective.¹² Therefore, further developments (implement ‘R’ sub-principles, reformulate questions, consider other FAIR initiatives, propose an archiving FAIR profile per data set, etc.) are planned to make an operational tool for public users. Later, DANS also produced *SATIFYD*¹³ a simple self-assessment tool based on 12 questions to sensibilise data set providers to the FAIR principles.

The GARDIAN FAIR metrics¹⁴ propose a list of indicators for calculating how much a CGIAR resource, publication, or data set meets the FAIR principles. The proposal defines 5 levels of compliance for ‘F’, ‘A’, and ‘I’, and it determines ‘R’ with a simple average calculation, i.e., ‘R’= ‘F’+‘A’+‘I’/3. For example, ‘F’-level 1 concerns data with ‘no PID and insufficient metadata and/or documentation,’ level 2 concerns data with ‘no PID but sufficient metadata and/or documentation,’ level 3 concerns data with ‘no PID but extensive metadata and/or documentation,’ level 4 concerns data with ‘PID and extensive metadata and/or documentation,’ and level 5 concerns data that ‘publish metadata in a CGIAR centre repository.’

In 2017, UPM-INIA, Oxford e-research centre and GO FAIR created the *FAIR Metrics* group and published the first ‘general, scalable, automatable FAIRness’ evaluation framework (Wilkinson et al., 2018, 2019), enabling any scientific community to define, implement and share metrics – called Maturity Indicators (MIs) – based on the community interpretation of the FAIR principles. MI creation or edition can be performed through specific templates on GitHub; they describe the MI metadata, e.g., identifier, name, to which data does it apply, what is being measured. Each MI is associated with a compliance test (implemented with the SmartAPI) to automatically check if a digital object meets the conditions of a MI; the scoring system is binary (pass/fail for each test). Once a YAML description of a metric is approved by the internal FAIR Metrics community, it becomes publicly available, ready for use and extendable for other kinds of digital objects. The FAIR Metrics initiative answers several challenges related to FAIR assessment with a framework for a variety of data services and components. It proposes a community-based collection of open MIs, tests, metadata and recommendations on how to improve FAIRness. However, we believe the evaluation framework should be extended with further improvements (such as a longitudinal record of the assessment scores for a given resource, integrating evaluation

into data management plans, etc.) and a peer-review approach to evaluate the MIs and make results more reliable. As mentioned in Section 5, in the future, we plan to adopt the FAIR Metrics framework to describe and implement our semantic resource FAIRness assessment methodology and externalise it from ontology repositories.

In 2018, the *OzNome 5-star tool* (Cox and Yu, 2022) was published by CSIRO. It is an online survey (containing multiple-choice questions) that generates a star chart representing the resultant degree of FAIR compliance of a data set according to some specific metrics. In addition to the FAIR principles, the authors also considered Berners-Lee’s 5-star linked open data https://www.semanticscholar.org/paper/Linked-Data-Bizer-Heath/c5a3894c3ebfd906b9ad0b2537475ffc2f5f84a8and treated aspects that are not covered by some specific tool for producing their data rating system, the FAIR self-assessment tool. Similarly, our methodology was influenced by the Linked Open Data principles because semantic resources and ontologies are frequently implemented using semantic web technologies.

In 2019, a *FAIRness assessment tool* for data librarians and information technology staff was proposed by ARDC as a series of questions related to each FAIR principle.¹⁵ It offers a green bar indicator that specifies the overall level of FAIRness of data sets. This prototype tool reflects ARDC’S interpretation of the FAIR principles, but, as mentioned by its authors, part of the proposed questions has been inspired by the FAIRdat and 5-star data rating tool. Based on our knowledge, no specifications about the scoring scheme are publicly available.

In 2020, the *FAIRshake* toolkit (Clarke et al., 2019) was proposed to help the community develop FAIR metrics. The toolkit enables a semi-automatic evaluation and offers the visualisation of FAIRness assessment as an insignia that can be embedded within digital-resources-hosting websites.

In 2020, *FAIR-Aware* (Devaraju et al., 2021) was proposed by DANS, DCC, and UniHB as an output of the FAIRsFAIR H2020 project. It is an online self-assessment questionnaire composed of 10 yes/no questions (3 for F., 2 for A., 1 for I. and 4 for R). Each question is associated with detailed information and links to assist users. FAIR-Aware still needs several improvements, such as offering a synthesised score, being compliant with FAIR Metrics, providing recommendations to enhance FAIRness as a result of using the questionnaire. The FAIRsFAIR project also developed the *F-UJI tool* (Devaraju and Huber, 2021),¹⁶ a web service for the FAIRness assessment of any digital object based on the FAIRsFAIR Data Object Assessment Metrics. The tool implementation follows the existing web standards such as re3data and DataCite APIs, SPDX License List, RDA metadata standards Catalogue and Linked Open Vocabularies (LOV) and PID resolution services best practices. FAIRsFAIR also formed several groups interested in the FAIRness assessment of specific digital objects such as web services or semantic artefacts (next section).

In 2021, the *FAIR cookbook*¹⁷ is proposed under the umbrella of the FAIRplus project. It is an online resource for the Life Sciences with exemplar recipes that aim to help
researchers in learning how to FAIRify their data sets. The current recipes are browsable, and new ones could be added and improved in an open manner.

In 2021, *FAIR checker* (Rosnet et al., 2021) is developed by the ELIXIR-France Interoperability Working group. The web service enables the evaluation of a digital resource, it exploits semantic web technologies to check if metadata use standards and recognised ontologies or controlled vocabularies. The tool follows the FAIR Metrics and includes detecting missing some minimal metadata for each FAIR sub-principle.

<figure><img src="image_2.png" /><figcaption>**Figure 1** Illustration of FAIRness assessment graphical visualisations from different tools: (a) *FAIRdat* uses a profile represented by 1 to 5 stars; (b) *FAIR self-assessment* uses a profile composed of blue-coloured letters per principle; (c) *FAIRshake* uses a grid coloured according to the level of the obtained score (from low scores in red to high scores in blue); (d) *FAIR checker* proposes a wheel visualisation showing the obtained results for each FAIR principle; (e) *FAIR-Aware* presents results with letters coloured according to the level of the obtained score (in blue for awareness and yellow for willingness)</figcaption></figure>

## 2.4 Ontology specific FAIRness assessment approaches

Before the FAIR principles, in 2011, Berners-Lee presented the foundational principles for *Linked Open Data* (Bizer et al., 2009) for making data available, shareable and interconnected on the web. The FAIR principles have been proposed for similar reasons with a stronger emphasis on data reusability (consideration of license agreement and

provenance information). The 5-stars LOD principles were specialised in 2014 for linked data vocabularies (Janowicz et al., 2014) as five rules to follow for creating and publishing ‘good’ vocabularies. Under this scheme, stars denote the quality of data leading to better structure (e.g., use of W3C standards) and interoperability for reuse (e.g., metadata description, reuse of vocabularies and alignment). The proposed 5-star rating system for vocabularies is simple; however, no implementation tool was developed for making the assessment automatic, and the principles are not largely referenced today. The degree to which the FAIR principles align and extend the 5-star open data principles was studied (Garijo and Poveda-Villalón, 2020; Hasnain and Rebholz-Schuhmann, 2018) and we have incorporated this alignment in our methodology. Note that another recommendation for publishing RDF vocabularies was also produced in 2008 by the W3C Semantic Web Deployment Working Group.¹⁸

In 2017, the *Minimum Information for Reporting an Ontology* initiative published the MIRO guidelines for ontology developers when reporting an ontology in scientific reports (Matentzoglu et al., 2018). The guidelines aim to improve the quality and consistency of the information content descriptions, including development methodology, provenance and context of reuse information. These guidelines refer to 34 information items (such as ‘ontology name,’ ‘ontology license,’ ‘ontology URL’) and specify the level of importance (must, should, optional) for each individual information item. This work was significant but was never aligned with or extended to the FAIR principles. However, the MOD 1.4 metadata model (next paragraph) provided an alignment between each MIRO guideline and the corresponding metadata properties in MOD. We have used this alignment in our methodology to influence the FAIRness assessment score with the MIRO guidelines.

Dutta et al. (2017) reviewed and harmonised existing metadata vocabularies and proposed a unified *Metadata for Ontology Description and Publication Ontology* (MOD)¹⁹ model to facilitate manual and automatic ontology descriptions, identification and selection. MOD is not another standard nor another metadata vocabulary, but more a set of identified properties one can use to describe a semantic resource.²⁰ MOD 1.4 was used in AgroPortal to implement a richer, unified metadata model (Jonquet et al., 2018a). AgroPortal recognises 346 properties from 15 relevant metadata vocabularies (Dublin Core, Ontology Metadata Vocabulary, VoID, FOAF, Schema.org, PROV-O, DCAT, etc.) and maps them to its unified model. This unified metadata model was the first step for enabling FAIRness assessment. For example, an ontology developer can focus on his/her responsibility of determining the license to use an ontology. At the same time, MOD and AgroPortal offer a means to encode such information in a way machines can use. Our assessment methodology relies on MOD suggested properties to implement tests as detailed in Sub-section 3.2.

In March 2020, the FAIRsFAIR H2020 project delivered the first version of a list of 17 recommendations and 10 best practices recommendations for making semantic artefacts FAIR (Le Franc et al., 2020). For each recommendation, the
authors provided a detailed description, a list of related supporting technologies or technical solutions. Similarly, best practices are introduced as recommendations not directly related to a FAIR principle but contribute to the overall evaluation of a semantic resource. This proposal is currently being discussed outside of the FAIRsFAIR projects, especially in the context of the RDA VSSIG. The recommendations are also publicly available for comments on GitHub.²¹ Our group is currently strongly involved in revising and commenting on the final version of the recommendations.

Later in 2020, Garijo and Poveda-Villalón (2020) considered some of the works already cited and produced ‘guidelines and best practices for creating accessible, understandable and reusable ontologies on the web.’ In another position paper, Poveda-Villalón et al. (2020) completed their work with a qualitative analysis of how four ontology publication initiatives cover the foundational FAIR principles. They propose some recommendations on making ontologies FAIR and list some open issues that might be addressed by the semantic web community in the future. These two publications are very relevant for our methodology; our work is a step further as it completes this work and proposes a concrete implementation for automatic assessment. In October, Garijo et al. (2021) proposed FOOPS! a web service for assessing an ontology regarding the FAIR principles. The tool is a good starting point for automatic assessment, and it adopts, like ours, an approach based on 24 questions/tests. Still, it has several limits: it is not reusing existing work covering all the sub-principles, it does not cover all the sub-principles (e.g., missing R1.3), it does not propose actionable guidelines to address the detected issues and it does not enable the assessment of a set of ontologies. One big difference is that FOOPS! does not depend/rely on any ontology repository nor a standard way to describe ontologies/metadata, which is somehow both an advantage and an inconvenient.

Other related work on FAIR principles for semantic resources exist but are not reused in our current methodology, include:

* A list of functional metrics and recommendations for Linked Open Data Knowledge Organisation Systems (LOD KOS) proposed in Zeng and Clunis (2020) for assessing the functionality against Functional, Impactful, Transformable (FIT) metrics and four recommendations for enhancing their FAIRness level (one recommendation per FAIR principle). This initiative is interesting, but the proposed preliminary recommendations are very limited; they do not cover all FAIR sub-principles e.g., Findable has only one recommendation that stresses the need ‘to enrich metadata as much as possible to enable data discovery’ without considering other aspects such as identifiers and libraries/repositories.

* Coxid et al. (2021) proposed guidelines (‘10 simple rules’) for making a vocabulary FAIR²² and transform vocabularies that are not available following web standards. The authors do not explain how the proposed rules are aligned to each FAIR sub-principle and do not consider any related initiatives.

* DBPedia Archivo (Frey et al., 2020) is an ontology archive released end 2020 to help developers and consumers implement FAIRer ontologies. At this moment, Archivo contains about 1032 ontologies. The prototype²³ automatically discovers, downloads, archives and rates new ontologies. Once an ontology is saved, Archivo determines its 4-stars FAIR rating, tracks its changes and updates its scores. This work highlights the role that ontology libraries and repositories play in the FAIRification process. Unfortunately, this work is not inspired by existing research methodologies/tools. We think that it needs to be improved in order to make its 4-star rating system clearer for the community.

<caption>**Figure 2** Formal FAIR score definition</caption>

$FAIRScore sr) = \sum_{i=1}^{4} FAIRPrincipleScore_i sr) = \sum_{j=1}^{n} FAIRSubPrincipleScore_{ij} sr) = \sum_{k=1}^{m} QScore_{ijk} sr)$ where n is the number of sub-principles associated with a principle, and m is the number of questions associated with a sub-principle. $QScore sr)$ is either the maximum number of credits or less depending how the sr passes the test. Scores are normalised using as:

$$ NormalisedScore sr) = (Score / Credits) * 100 $$

where the credits are detailed in the grid proposed in Garijo and Poveda-Villalón (2020).
# 3 FAIRness assessment methodology

## 3.1 FAIR scores specification and evaluation

FAIRness assessment is about constructing a scoring function, here called *FAIRore*, which for each semantic resource (*sr*) returns a numeric value representing the level of FAIRness. Formally, *FAIRSore(st)* is the sum of four *FAIRPrincipleScore* obtained for each principle themselves by suming *FAIRSubPrincipleScore* which are the sum of a series of *QSore* obtained for each assessment question (see Figure 2).

To determine the appropriate credits for each FAIR sub-principle, we analysed and merged relevant related approaches cited in Sub-section 2.4, namely FDMM (v0.04), SHARC (v1.1), LOD 5-stars V, MIRO, FAIRsFAIR recommendations and Poveda-Villalón et al.'s (2020) guidelines. We consider both generic and specific approaches to provide a specialised methodology for ontologies still being influenced by more general concerns, as ontologies are a kind of digital object. The integration was not straightforward because none of the approaches used is simply and strictly aligned with the 15 sub-principles (e.g., FDMM provides 47 criteria), and two of them (i.e., MIRO and 5-stars V) were totally disconnected from the FAIR prism. The outcome of this alignment work is an integrated FAIRness assessment grid (Amdouni and Jonquet, 2021) of 478 credits that enables the evaluation of any semantic resource by creating a metric – and thus possible thresholds (Figure 3, Annex 1). The grid is conceived in a way that can be customised, extended or improved by other semantic experts in further studies. Next section, we explain how we dispatch the grid's credits on a series of 61 questions to assess each FAIR sub-principle in the O'FAIRe methodology.

## 3.2 FAIRness assessment questions and relevant metadata properties

In this section, we interpret each FAIR sub-principle for semantic resources and list standardised metadata properties that could be used to encode the information necessary to

address the sub-principle. For each FAIR sub-principle, we propose a series of assessment questions and identify the MOD 1.4 properties to use to answer these questions. MOD allows us to determine which property can be used unambiguously; however, our methodology is independent of MOD and only requires that information is represented in ontology metadata in whatever way.

Over the 61 questions we described two kinds of metadata: ‘core’ and ‘extra’ metadata properties. Referring to MOD1.4, we have identified 57 ‘core’ metadata properties allowing 276 credits over a total of 478. To determine the list of extra metadata properties, we separate the metadata properties for any principles from those for F2, which has to be treated apart. Indeed, F2 (‘Data are described with rich metadata’) was assigned all the properties that MOD1.4 include as relevant for ontologies but have not been assigned to another sub-principle (71 ‘extra’ metadata properties). The idea is that any ontology using some of the ‘extra’ metadata properties in addition to the core 57 ones, will be ‘FAIRer’.

In what follows, we rephrase the original FAIR principles with the word ‘ontologies’ (for *data*) and ‘*ontology metadata*’ (for *metadata*). This helps ‘projecting’ the FAIR principles to ontologies. We detail what this sub-principle means or implies for ontologies and ontology metadata. And for each sub-principle, we list in boxes the questions used for FAIRness assessment and the number of credits O’FAIRe will attribute if the test is valid. For each sub-principle, we have assigned every question a certain number of credits so that the total will correspond to the number of credits identified in Figure 3 (478 total). The distribution of the credits on each question is done as much as possible ‘equally & logically’ but may be biased by our interpretation of how to make semantic resources FAIR; however, it can easily be amended or adjusted if questions are removed/added in the future. When possible, for each sub-principle, we also briefly mention how it links to related work. As much as possible, we provide examples of how specific ontologies or semantic resources address a sub-principle. And if a sub-principle is addressed by the ontology repository or library, we provide examples from AgroPortal.

<figure><img src="image_3.png" /><figcaption>**Figure 3** Integrated quantitative FAIRness assessment grid</figcaption></figure>
F1. Ontologies and ontology metadata are assigned a globally unique and persistent identifier: F1 is about identifiers; although it was not mentioned in the original principle expression, several studies argued that identifiers must also be resolvable, i.e., the identifier must link back to access to the resource or the resource description. FAIRsFAIR recommendations refer to the concept of *Globally Unique, Persistent and Resolvable Identifier (GUPRI)* (Frasson and Kostopoulos) and proposed adopting it for both ontologies and ontology metadata. Typically, ontologies described with semantic web languages are assigned a *Unique Resource Identifier (URI)*, and metadata are either included within the ontology file or edited in a separate file. URIs are usually globally unique, but their persistent and resolvable characteristics are not guaranteed. In some cases, ontologies can be assigned an additional *Globally Unique IDentifier (GUID)* assigned by an external organisation such as a DOI. Sometimes, URIs take the form of *Persistent URLs (PURLs)* that are committed to be persistent over time. Ideally, ontology URIs should be resolvable, i.e., link back to the actual resource in a relevant format to ensure a higher degree of compliance for F1.²⁴ When stored in a separated file, the same rules – i.e., using GUPRIs – must apply to ontology metadata file(s) as detailed later (F2 and F3).

In addition, the ontology community sometimes tries to maintain a coherent use of acronyms to identify ontologies. For instance, in the OBO Foundry²⁵, a mandatory short name is assigned and used for human-friendly identification of the ontology and PURL assignment.

With the MOD ontology metadata model, F1 can be assessed by checking the value assigned to the property *owl:ontologyIRI*, used to encode the ontology's URI and the property *dct:identifier*, used to encode another 'external' identifier. In addition, the property *owl:versionIRI*, which stores a version-specific URI, can also be used to evaluate if the ontology clearly distinguishes version identifiers. Ontology acronyms (encoded in MOD with *omv:acronym*) are useful information. Still, they cannot be used for FAIRness assessment as they do not respect any of the required attributes for GUPRIs. The next box summarises the evaluation questions regarding the different types of ontology and metadata identifiers cited above – resolvability will be evaluated in A1 – and specifies the proposed distribution of credits:

<table><thead><tr><td>F1 assessment questions (41 credits)</td><td>Identifiers</td></tr></thead><tbody><tr><td>Q1. Does the ontology have a 'local' identifier, i.e., a globally unique and potentially permanent identifier assigned by the developer (or developing organisation)? <b>9 pts</b></td><td></td></tr><tr><td>Q2. Does the ontology provide an additional 'external' identifier, i.e., a guarantee globally unique and persistent identifier assigned by an accredited body? If yes, is the external identifier a DOI? <b>11 pts</b></td><td></td></tr><tr><td>Q3. Are the ontology metadata clearly identified either by the same identifier as the ontology (if included in the ontology file) or with its own globally unique and persistent identifier? <b>12 pts</b></td><td></td></tr><tr><td>Q4. Does the ontology provide a version-specific URI, and is this URI resolvable? <b>9 pts</b></td><td></td></tr><tr><td>Related recommendations: (i) FAIRsFAIR: P-Rec1 and 2; (ii) Poveda el al.: Rec 1, 2, and 3.</td><td></td></tr></tbody></table>

Examples:

* The Animal Trait Ontology for Livestock (ATOL) is identified by its URI (*owl:ontologyIRI* = http://opendata.inra.fr/ATOL) but in addition has been assigned by INRAE's official data repository a specific DOI (*dct:identifier* = 10.15454/1.4690062322351956E12) for long term identification of the resource. ATOL's metadata are included in the main ontology file.

* The AGROVOC thesaurus is identified by its URI (*owl:ontologyIRI* = http://aims.fao.org/aos/agrovoc) but no other external identifier. AGROVOC's metadata are not included in the main file but live aside in their own VoID file and identified with its own URI (http://aims.fao.org/aos/agrovoc/void.ttl).

* The Cell Ontology has been assigned a short name (CL) and corresponding PURL as URI by the OBO-F (*owl:ontologyIRI* = *dct:identifier* = http://purl.obolibrary.org/obo/cl.owl), and no other identifier is needed as the PURL respects GUPRIs requirements. CL's metadata is partly included in the ontology file but also aside in a YAML document on GitHub as encouraged by OBO-F.

* The Environment Ontology (ENVO) provides a version specific URI for each version (e.g., *owl:versionIRI* = http://purl.obolibrary.org/obo/envo/releases/2021-05-14/envo.owl). The ontology URI (*owl:ontologyIRI* = http://purl.obolibrary.org/obo/envo.owl) always resolves to the latest version specific URI, which itself resolves to the actual ontology versioned file.

F2. Ontologies are described with rich ontology metadata: F2 criterion states the importance of describing ontology with an information model that formalises their descriptions. FAIRsFAIR (P-Rec 2) recommends using 'common minimum metadata schema to describe semantic artefacts and their content,' and the project is currently leading an activity to identify the relevant metadata properties for such a minimal model. FAIRsFAIR recommends the use of relevant metadata vocabularies and schema such as MOD, the *Ontology Metadata Vocabulary (OMV)* (Suarez-Figueroa et al., 2005) and the *Linked Open Vocabularies (LOV)* (Vandenbussche et al., 2014). With all the metadata properties being previously reviewed and aggregated within MOD, we know focusing on MOD allows us to respect this recommendation. F2 implies the definition of intrinsic (e.g., file format, creation date, status, syntax, etc.) or user-defined contextual information (e.g., title, description, language, comments, projects, relations, etc.). Intrinsic metadata can be either directly provided by the author of the ontology during the creation process or automatically generated. Several approaches for ontology metadata description were extensively reviewed by Jonquet et al. (2018a). We believe that an ontology metadata model should as much as possible: (i) be based on relevant existing metadata vocabularies (e.g., DCAT, DC, OMV, PROV-O, etc.), especially W3C Recommendations and (ii) help to implement recommendations and guidelines such as the ones reviewed Sub-section 0.
F2 can be assessed by evaluating the quantity and quality of the metadata properties used to describe an ontology. MOD recognises 346 properties from 23 metadata vocabularies and groups them in an ontology metadata model of 128 properties. Because our FAIRness assessment methodology is mostly based on qualitative metadata evaluation, many metadata properties will be used to assess other specific FAIR sub-principles – such as the three properties listed above for F1 and the rest of this section. Consequently, when assessing F2, we will focus on metadata properties from the MOD model that are not used to assess another FAIR sub-principle. As some properties are clearly recognised as more important/informative than others, we will rely on the MIRO qualifications ('must,' 'should' and 'optional') as factors for every remaining property in MOD. The complete list of 71 MOD1.4 metadata properties used to assess F2 is provided in Annex 2 with alignment with the MIRO guidelines. Some of these proposed metadata are also recommended by Poveda-Villalón et al. (cf. Rec 6 about ontology metadata) such as `dct:title`, `dct:description`, `dct:abstract`, `dct:created`, `dct:subject`, `dct:modified`, `foaf:page`, `foaf:logo`, `foaf:depiction`, `owl:backwardCompatibleWith`, `vann:preferredNamespace`.

<table><thead><tr><td>F2 assessment questions (27 credits)</td><td>Metadata</td></tr></thead></table>

**Q1.** Is the ontology described with additional 'MIRO must' metadata properties? **16 pts**

**Q2.** Is the ontology described with additional 'MIRO should' or 'optional' metadata properties? **4 pts**

**Q3.** Is an ontology described with another metadata property with no explicit corresponding MIRO requirement? **7 pts**

**Related recommendations:** (i) FAIRsFAIR: P-Rec 3; (ii) Poveda et al.: Rec 6.

### Example:

* ENVO OWL source file contains 12 metadata properties from multiple metadata vocabularies: OWL, DC, DCT, DOAP, OboInOwl, RDFS, FOAF. 5 of these properties may be used to assess other sub-principles, however 7 'extra' properties may be used to assess F2: `dct:title`, `dct:description`, `foaf:homepage`, `doap:repository`, `doap:bugDatabase` as 'MIRO must', `oboInOwl:default-namespace` (mapped by MOD to `vann:preferredNamespacePrefix`) as 'MIRO optional' and `rdfs:comment` which has no mapping to MIRO.

**F3. Ontology metadata clearly and explicitly include the identifier of the ontologies they describe:** F3 criterion requires the association between an ontology and its metadata to be explicit by mentioning the ontology GUPRI in the metadata file and vice-versa. If the ontology metadata are included in the ontology file, no additional linking is required, and this sub-principle is automatically fulfilled. This sub-principle is especially relevant when ontology metadata are stored in another file or record; then, an explicit property needs to link an ontology to its metadata.

F3 can be assessed by differencing if the ontology metadata is included in the ontology file and, if not, by identifying the ad-hoc link to the ontology. As in F1, our methodology does not favour any approaches (i.e., Q1 is the same as Q2+Q3). However, we think a clarification is needed regarding the situations in which ontology publishers provide metadata in a separate file. To the best of our knowledge, there are no existing metadata vocabularies that define a property to relate an object to its metadata. None of the 23 metadata vocabularies reviewed when building MOD offers such a property. This issue is currently being addressed by the FAIR Digital Object framework work supervised by GO FAIR.²⁶ Eventually, the `fdo:hasMetadata` and `fdo:isMetadataOf` properties will be soon available.

<table><thead><tr><td>F3 questions (21 credits)</td><td>Link ontology-metadata</td></tr></thead><tbody><tr><td>Q1. Are the ontology metadata included and maintained in the ontology file? **21 pts**</td><td></td></tr><tr><td>Q2. If not, are the ontology metadata described in an external file? **11 pts**</td><td></td></tr><tr><td>Q3. Does that external file explicitly link to the ontology and vice-versa? **10 pts**</td><td></td></tr><tr><td>Related recommendations: (i) FAIRsFAIR P-Rec 2.</td><td></td></tr></tbody></table>

### Examples:

* AGROVOC keeps metadata aside in a specific VoID profile, which explicitly links back to the thesaurus with the `dct:source` metadata property.

* OBO Foundry ontologies are described in GitHub by YAML files;²⁷ the link between an ontology and its corresponding metadata file and vice-versa is explicit (i.e., each ontology has its own file in GitHub called `ONT.md`) but not semantically represented with a metadata vocabulary property. For example, the Phentotype and Trait Ontology (PATO) metadata are available in `pato.md` file of the ontology.²⁸

**F4. Ontologies and ontology metadata are registered or indexed in a searchable resource:** A searchable resource for ontologies and ontology metadata means an application and endpoint supporting several means of querying metadata or ontology content, e.g., term and relation lookup (via identifier or text-based). This aspect is particularly emphasised by several FAIRsFAIR's recommendations addressed to repositories, e.g., P-Rec 4: 'Semantic artefact and its content should be published in a trustworthy semantic repository' but also P-Rec 5,6,7. FAIRsFAIR identifies the NCBO BioPortal (Noy et al., 2009), EBI-OLS (Côté et al., 2006), EcoPortal (Fiore et al., 2017), AgroPortal (Jonquet et al., 2018b), LOV (Vandenbussche et al., 2014), BODC NERC vocabulary service or more generic services such as Finto.fi, BARTOC or Research Vocabularies Australia (see Annex 3 for URLs). Both FAIRsFAIR (i.e., P-Rec 5, 6) and Poveda-Villalón et al. (i.e., Rec 10 about 'making an ontology findable on the web (Garijo and Poveda-Villalón, 2020) argue F4 is better realised if 'semantic repositories' are properly indexed by web search engines.
In the literature, ‘ontology library’ include systems that help reuse or discover ontologies by simply listing them (e.g., DAML, Protégé or DERI listings) or by offering structured metadata to describe them (e.g., FAIRSharing, OBO Foundry, BARTOC) but no service based on the content of the ontologies. The expression ‘ontology repository’ (Hartmann et al., 2009) refers to web applications (e.g., BioPortal, EBI-OLS, AgroPortal, Ontobee) with advanced content-based features (search, browsing, metadata management, visualisation, mappings) accessible via user interfaces and application programming interfaces. Other terms are used in the literature to identify such resources, such as ontology registry, collection, or terminology service, or semantic repository (Jonquet, 2022). In our methodology, we consider both ontology libraries and repositories but distinguish them to assign scores.

F4 can be assessed by counting in how many libraries (metadata indexing) or repositories (data indexing) an ontology is hosted. MOD property `schema:includedInDataCatalog` can be used to store in which library/repository a semantic resource is included. Annex 3 lists the repositories and libraries currently used in O'FAIRe. One community can also evaluate the importance of being present in this or that ontology library/repository; however, in O'FAIRe, to keep a tool as generic as possible, we have not implemented any community-specific evaluation aspect (except in R1.3). In addition, the quality of indexing the ontology metadata (resp. content) inside an ontology library (resp. repository), as well as the quality of indexing of such libraries/repositories by external web search engine (cf. Q3), are information that are not specific to the ontology but directly depend on the libraries/repositories. Those aspects are not evaluated yet in O'FAIRe prototype presented Sub-section 0.

F4 assessment questions (24 credits)

Repository

Q1. Is the ontology registered in multiple ontology ‘libraries’? **6 pts**

Q2. Is the ontology registered in multiple open ontology ‘repositories’? **10 pts**

Q3. Are the ontology ‘libraries’ or ‘repositories’ properly indexed by web search engines? **8 pts**

**Related recommendations:** (i) FAIRsFAIR P-Rec 4, 5 and 6 (ii) Poveda et al. Rec 10.

Examples:

* The Comparative Data Analysis Ontology (CDAO, http://purl.obolibrary.org/obo/cdao.owl) is available in several open ontology repositories (e.g., BioPortal, AgroPortal, EBI-OLS) and also in multiple ontology libraries such OBO-F, FAIRSharing, and Agrisemantics Map of Data Standards.

* AgroPortal is a reference ontology repository in the agri-food domain based on the generic and open source OntoPortal technology (https://ontoportal.org) (Graybeal et al., 2019). AgroPortal allows users to search for metadata and ontology content; both are stored in an RDF triplestore. The system also offers a Lucene index text-based search.

AgroPortal and its content are very well indexed and retrievable directly via web search engines which make ontologies even more findable. Plus, for an agri-food ontology, being in AgroPortal means that the ontologies will be automatically registered in FAIRsharing and Agrisemantics Map of Data Standards.

A1. Ontologies and ontology metadata are retrievable using a standardised communication protocol by their identifier: A1 criterion expresses the need for a mechanism enabling to retrieve – i.e., obtain the files – ontologies and ontology metadata by their identifier(s) (cf. F1). In the semantic web world, the standard communication protocol is HTTP paired with URIs. An ontology file hosted on a web server becomes accessible via HTTP, a communication protocol standardised by W3C. Plus, if the URIs of an ontology are resolvable, objects of the ontology (including itself, its metadata and its content) will be distinguishably retrievable by HTTP. Poveda-Villalón et al.’s Rec 9 requires also URIs to be resolvable with content negotiation, i.e., resolve to different formats or destinations depending on the request.

This is typically the role of repositories to provide access to ontologies or ontology metadata via HTTP and/or web services API. An ontology repository can be the destination for resolving URIs to web pages if it provides a specific landing page for each ontology object. For instance, OBO Foundry PURLs resolve to OntoBee or AGROVOC URIs resolve to AIMS’s SKOSMOS instance. If ontologies and ontology metadata are assigned other identifiers, they should be also resolvable, e.g., a DOI becomes resolvable once prefixed by ‘https://doi.org/’.

A1 can be assessed by verifying if ontology and ontology metadata URIs or other identifiers, if exist, are resolvable via HTTP and, if they support content negotiation. See F1 for the questions/properties used to assess the existence of identifiers. Here, we go further and verify the resolvability of the identifiers. We can also evaluate if other standardised communication protocols are supported, e.g., a REST or SOAP web service API or a SPARQL endpoint. The property `sd:endpoint` can be used to store the SPARQL endpoint that can be used to retrieve ontology content and metadata.

A1 questions (43 credits)

Resolvable identifiers

Q1. Do the ontology URI and other identifiers, if they exist, resolve to the ontology? **6 pts**

Q2. Does the ontology URI (if metadata are included in the ontology file) or the external metadata URI resolves to the metadata record? **7 pts**

Q3. Do the ontology URI and the external metadata URI (if the metadata are not included in the ontology file) support content negotiation? **24 pts**

Q4. Are the ontology and its metadata accessible through another standard protocol such as SPARQL? **6 pts**

**Related recommendations:** (i) FAIRsFAIR P-Rec 5 (ii) Poveda-Villalón et al. Rec 7,9 (iii) 5-stars V principles 1, 2, 4.

Examples:

* ENVO provides URIs as resolvable PURLs: e.g., http://purl.obolibrary.org/obo/ENVO_00000133 will resolve to an OntoBee landing page.
* EDAM ontology (http://edamontology.org) provides a negotiable URI making the ontology available in multiple formats: HTML, RDF/XML and OBO.²⁹

* The Agri-Food Experiment Ontology³⁰ provides an additional external identifier (a DOI): https://doi.org/10.15454/DPBMBW, which resolves to a data repository hosting the ontology.

* The ANAEE thesaurus³¹ metadata in AgroPortal is retrievable in different formats such as JSON-LD.³²

* Ontologies hosted in AgroPortal are all described with a value for `sd:endpoint`, which is either AgroPortal's SPARQL endpoint or a specific endpoint provided by the ontology developer. For instance, TAXREF-LD taxonomy has `sd:endpoint=http://taxref.mnhn.fr/sparql`.

A1.1. The protocol to retrieve ontologies and ontology metadata is open, free, and universally implementable: A1.1 criterion refines requirements for the communication protocol. As stated in A1, semantic web technologies are based on HTTP/URI, which meet each of these requirements. Ontology repositories and libraries are mostly web applications/databases which natively implement HTTP access to ontologies and ontology metadata. They can also support access via other open, free and universally implementable communication protocols such as FTP or query endpoints such as SPARQL.

A1.1 assessment is very much linked to A1. We focus here on the use of HTTP and on the use of other open, free and universally implementable protocols.

A1.1 assessment questions (28 credits)

Protocol

**Q1.** Is the ontology relying on HTTP/URIs for its identification and access mechanisms? **20 pts**

**Q2.** Is the ontology access protocol open, free, and universally implementable? **4 pts.**

**Q3.** If the ontology and metadata are accessible through another protocol, is that protocol open, free, and universally implementable? **4 pts.**

Related recommendations: (i) FAIRsFAIR P-Rec 5.

A1.2. The protocol to retrieve ontologies and ontology metadata supports authentication and authorisation when an ontology has access restriction: A1.2 criterion continues to refine requirements for the communication protocol here to ensure secure access when needed. Indeed, in some cases, ontologies or semantic resources are not publicly available and require access or licensing restrictions. In that case, their URIs are usually not resolvable or rely on HTTPS for authentication and use a web server that supports authorisation. Note that ontology metadata can be publicly accessible or not independently of the ontology content itself. Typically, this is the role of an ontology libraries/repositories to provide open access to metadata and to serve ontologies privately when they support user/group accounts (authentication) and enable access to particular users (authorisation). Here again, the use of an ontology repository facilitates the realisation of A1.2. This is explicitly mentioned in FAIRsFAIR P-Rec 7 ('Administrators should offer a secure protocol and user access control').

A1.2 can be assessed by evaluating if the resolution of identifiers (A1) supports authentication and then authorisation. Additionally, the MOD model suggests using the property `dct:accessRights` to describe access restrictions. We can also assess if ontologies and ontology metadata are accessible in repositories that support authentication and authorisation. In O'FAIRe implementation, we have chosen the second solution.

A1.2 assessment questions (22 credits)

Protocol

**Q1.** Is the ontology accessible through a protocol that supports authentication and authorisation? **11 pts**

**Q2.** Are the ontology metadata accessible through a protocol that supports authentication and authorisation? **11 pts**

Related recommendations: (i) FAIRsFAIR P-Rec 7.

### Examples:

* BioPortal and AgroPortal both support authentication and authorisation: a user must create an account to upload ontologies or edit ontology metadata, and the portals support user profiles (e.g., librarian, administrator). Access to private ontologies can be granted on a case-by-case approach. Additionally, ontologies in BioPortal or AgroPortal can be either fully private or semi-public, i.e., browsable and queryable but not publicly downloadable. For instances:

* The Food classification and description system (FOODEX2, http://data.food.gov.uk/codes/foodtype) provided as RDF is a private resource in AgroPortal, not openly accessible.

* The Medical Dictionary for Regulatory Activities (https://bioportal.bioontology.org/ontologies/MEDDRA) is browsable and accessible via API in the NCBO BioPortal, but the ontology cannot be downloaded.

A2. Ontology metadata are accessible even when the ontology is no longer available: A2 criterion argues for a clear separation of an ontology and its metadata to ensure long-term availability of the metadata. Again, to fulfil A2, the role of ontology repositories or libraries, if permitted by the ontology authors, is to ensure access to archived ontologies and their metadata. Alternatively, it should serve only the metadata if the ontology itself is no longer available. This aspect is also identified by FAIRsFAIR P-Rec 8, which indicates that an ontology repository or library should offer human and machine-readable persistence policies to save ontology and ontology metadata changes. Added to this, we believe that an ontology repository must support versioning and allow publishing ontologies during the different production phases (i.e., beta, production, etc.) and clearly inform about the status of ontologies, especially if it becomes obsolete or deprecated.

A2 can be assessed by checking if all (or a significant number of) ontology versions are versioned and archived. We can also assess if ontologies are accessible in repositories or libraries that support metadata archiving. In addition, the MOD model offers the properties `omv:status` to indicate the different production phases (alpha, beta, production,
retired) and `owl:deprecated` as a Boolean to indicate if the ontology is deprecated.

<table><thead><tr><th>A2 questions (20 credits)</th><th>Metadata long term access</th></tr></thead><tbody><tr><td><p><strong>Q1.</strong> Is the ontology accessible in a repository that supports versioning? <strong>7 pts</strong></p><p><strong>Q2.</strong> Are the ontology metadata of each version available? <strong>5 pts</strong></p><p><strong>Q3.</strong> Are the ontology metadata accessible even if no more versions of the ontology are available? <strong>4 pts</strong></p><p><strong>Q4.</strong> Is the status of the ontology clearly informed? <strong>4 pts</strong></p><p><strong>Related recommendations:</strong> (i) FAIRsFAIR: P-Rec 8.</p></td><td></td></tr></tbody></table>

Examples:

* OBO Foundry suggest best practices for versioning (when/how creating a new file) and rely on GitHub for versioning content and metadata files.

* BioPortal and AgroPortal archives all the versions of the ontologies (and their metadata) previously hosted on the repository. Each version is automatically numbered and can be downloaded in different formats; however, only the content of the latest version is accessible via the repository services such as browsing, navigating, search and annotation.

* AgroPortal supports access to rich ontology metadata even when an ontology is no longer available.

I1. *Ontologies and ontology metadata use a formal, accessible, shared and broadly applicable language for knowledge representation:* I1 criterion emphasises the importance of the knowledge representation language, an aspect identified by most existing recommendations. An ontology is typically a digital resource that relies on a formal language built to be as much as possible machine-understandable. However, some ontologies or semantic resources might – for miscellaneous reasons – be described in textual (verbose) or graphical (image) form, which are not directly usable by a machine. Typically, an ontology is stored in a file using a dedicated syntax (RDF/XML, Turtle, JSON-LD) and representation language (OWL, SKOS, RDFS, OBO).³³ A semantic resource or artefact may have different levels of formality (e.g., ontology, terminology, thesaurus, vocabulary, etc. as, e.g., listed in the NKOS types vocabularies³⁴ or OMV (Suarez-Figueroa et al., 2005)) to which some representation language better correspond. Ontology metadata are usually represented using the same syntax and representation language as the ontology itself. However, this assumption is not always true when the metadata are stored aside from the ontology; thus, the language for knowledge representation must be assessed independently. Note that ontologies and ontology metadata files should be parsed without any issues using a parser adapted to their syntax and language.

I1 can be assessed by evaluating the level of formality and accessibility – in the sense easily understandable – of the representation language as well as how much it is shared/adopted by a community, and still broadly applicable (i.e., not task nor domain-specific) and recommended by relevant standardisation bodies (i.e., W3C). In our methodology, we have considered semantic resources, and

their metadata can be described using OWL, OBO, RDFS and SKOS representation language (independently of their syntax) as well as in CSV, XML, PDF or TXT formats. Our classification of these ‘description formats’ is detailed in Annex 4. Typically, OWL is the most formal and broadly used representation language, but it is not the most accessible. For the assessment of I1, we can also evaluate the availability of the syntax descriptions and the level of formality, as well as the existence of distributions in other formats/syntaxes. In the MOD model, the representation language of an ontology is represented via the property `omv:hasOntologyLanguage`, its level of formality is described with the property `omv:hasFormalityLevel`, its syntax is described with the property `omv:hasOntologySyntax`. If a semantic resource distribution is available in another format or syntax, this information can be described with the properties `dct:hasFormat` and `dct:isFormatOf`.

<table><thead><tr><th>I1 questions (44 credits)</th><th>Knowledge representation</th></tr></thead><tbody><tr><td><p><strong>Q1.</strong> What is the representation language used for the ontology and ontology metadata? (*) <strong>20 pts</strong></p><p><strong>Q2.</strong> Is the representation language using a W3C Recommendation? <strong>10 pts</strong></p><p><strong>Q3.</strong> Is the syntax of the ontology informed? <strong>5 pts</strong></p><p><strong>Q4.</strong> Is the formality level of the ontology informed? <strong>5 pts</strong></p><p><strong>Q5.</strong> Is the availability of other syntaxes/formats informed? <strong>4 pts</strong></p><p>(*) Scoring scale of each representation format (cf. Annex 4): (OWL, 18 pts), (SKOS, 15 pts), (RDFS, 12 pts), (OBO, 11 pts), (XML, 10 pts), (CSV, 9 pts), (TXT, 7 pts), (PDF, 5 pts).</p><p><strong>Related recommendations:</strong> (i) FAIRsFAIR P-Rec 9, 10, 11 and 12; (ii) Poveda et al. Rec 6 and 9; (iii) 5-stars V principle 2.</p></td><td></td></tr></tbody></table>

Example:

* The Plant Experimental Conditions Ontology (PECO) is described in OWL (http://purl.obolibrary.org/obo/peco.owl) but is also available in OBO (http://purl.obolibrary.org/obo/peco.obo), its formality level is ‘ontology’ (http://w3id.org/nkos/nkostype #ontology) and its syntax is (http://www.w3.org/ns/formats/RDF_XML) RDF/XML. The AgroPortal entry for PECO rigorously describes each of these aspects using MOD and URIs for metadata values.

I2. *Ontologies and ontology metadata use vocabularies that follow FAIR principles:* For any kind of data or metadata, I2 criterion emphasises the use of semantic resources (called here with the generic term ‘vocabularies’) that are themselves FAIR.³⁵ I2 is the main motivation of our work on FAIRness assessment of semantic resources. However, if we consider semantic resources are themselves data to which the FAIR principles apply, then I2 emphasises the importance of reusing/relying on other vocabularies. In the realm of semantics, ontologies are encouraged to reuse other vocabularies in different manners (as listed in Laadhar et al. 2020a):

* formal imports (i.e., using the `owl:imports` construct), which result in every statement in the imported ontologies being present in the importing one;
* object reuses by simply reusing URIs from another ontology (assuming no new conflicting statements);

* alignments by creating qualified mappings to external URIs representing the same (or similar) objects.

One developer can also be influenced by another semantic resource, without making any explicit references to its objects. In all cases, the different kinds of relations between the semantic resource reusing and the one being reused shall be encoded in the metadata.

Ontology metadata reuse of vocabularies is more straightforward; it consists of (i) reusing a metadata property from an external metadata vocabulary (e.g., `dc:creator`, `cc:license`, `omv:hasNaturalLanguage`) and/or (ii) assigning values to a metadata property coming from another vocabulary (e.g., https://orcid.org/0000-0003-3934-0072, https://creativecommons.org/licenses/by/4.0, http://lexvo.org/id/iso639-3/eng). This latest aspect will be covered by I3. In both cases, when reusing external vocabularies, ontologies and ontology metadata should prefer open, formally recommended (e.g., W3C Recommendations, ISO, Dublin Core standards) or community-accepted vocabularies. By reusing a standard vocabulary, it plays the role of a common denominator to make ontologies and their metadata more interoperable.

I2 can be assessed by evaluating the proportional quantity (i.e., counting the number) and type of reuses of other vocabularies and by verifying the resolution of the imported/reused/aligned objects or influential vocabularies. We believe that it is important that each community define its preferred foundational ontologies (e.g., BFO, DOLCE), metadata vocabularies (DC, DCAT, PROV) or reference standard model (e.g., QUDT, OBOE, SSN/SOSA) as well as conceptual mappings guidelines (e.g., SKOS mapping properties, `owl:sameAs`, `oboInOwl:hasDbXref`). Plus, when looking at reuses one can also evaluate if the object reused are well-formed, e.g., do they include the minimum information for those terms such as suggested by MIREOT (Courtot et al., 2011) or assess if the alignments are well represented (as discussed for example in Laadhar et al. (2020b)). The MOD metadata model does provide 20 properties to describe relations between ontologies including for imports (`omv:useImports`), reuses (`door:ontologyRelatedTo`), alignment (`door:isAlignedTo`), influence (`door:explanationEvolution`, `voaf:similar`, `voaf:generalizes`, `schema:translationOfWork`) and reuse of metadata vocabularies (`voaf:metadataVoc`). The metadata vocabularies currently checked by O'FAIRRe are listed in Annex 5 with a coarse grain '4-level based FAIRness coefficient' manually assigned.

The FAIRness of reused vocabularies – if they are resolvable – has itself to be assessed using a methodology similar to the one presented in this article and will influence the FAIRness level of the semantic resources reusing these vocabularies. To the best of our knowledge, no metadata vocabularies nor the MOD metadata model, provide a 'FAIRness score' property that could be used to store the level of FAIRness of the reused vocabularies, which could

obviously create a chicken and egg issue in the assessment of I2. In the future, we will rely on the AgroPortal O'FAIRRe prototype (Sub-section 0) when reused vocabularies are themselves hosted in AgroPortal and on a fixed user self-assigned value when not. We will be extending MOD to include a specific metadata property to store FAIRness values and I2 related aspects such as alignment curation state (referred by Q5 and Q6).

<table><thead><tr><td>12 questions (32 credits)</td><td>FAIR vocabularies</td></tr></thead><tbody><tr><td>Q1.</td><td>Does the ontology import other FAIR vocabularies? <b>5 pts</b></td></tr><tr><td>Q2.</td><td>Does the ontology reuse terms from other FAIR vocabularies (URIs)? <b>5 pts</b></td></tr><tr><td>Q3.</td><td>If yes, does it include the minimum information for those terms? <b>3 pts</b></td></tr><tr><td>Q4.</td><td>Is the ontology aligned to other FAIR vocabularies? <b>5 pts</b></td></tr><tr><td>Q5.</td><td>If yes, are those alignments well represented and to unambiguous entities? If yes, are those alignments curated? <b>7 pts</b></td></tr><tr><td>Q6.</td><td>Does the ontology provide metadata information about the relation to or influence of other FAIR vocabularies? <b>2 pts</b></td></tr><tr><td>Q7.</td><td>Does the ontology reuse standard and FAIR metadata vocabularies to describe its metadata?? <b>5 pts</b></td></tr><tr><td colspan="2">Related recommendations: (i) FAIRsFAIR P-Rec 10 and 14.</td></tr></tbody></table>

### Examples:

* The Flora Phenotype Ontology (FLOPO, http://purl.obolibrary.org/obo/flopo.owl) explicitly imports the Plant Ontology using an `owl:imports`.

* The Agronomy Ontology (AGRO, http://purl.obolibrary.org/obo/agro.owl) is built by explicitly reusing terms from ENVO, CheBI and PATO. The developer uses the ROBOT software (Jackson et al., 2019) to materialise correct imports when building the ontology file.

* The Vertebrate Trait Ontology (VT) is declared in AgroPortal (http://agroportal.lirmm.fr/ontologies/VT) to be explicitly aligned to the Animal Trait Ontology for Livestock (ATOL) but also related to the Livestock breed Ontology (LBO) and to the Livestock Product Trait (LPT) ontology.

I3. Ontologies or ontology metadata include qualified references to other (meta)data: I3 criterion refers to the assessment of qualified references; references to vocabularies are covered by I2, thus we will focus in I3 on references to other types of data (e.g., database or a database element) that we call cross-references or annotations. Cross-references are generally used to indicate an ontology object corresponding element in a (public/open) database, e.g., the XRef (`oboInOwl:hasDbXref`) properties for terms in the OBO world (Laadhar et al., 2020). Annotations can be understood in a broader sense, informing that an ontology term annotates/describes a database or database element. For ontology metadata, a reference to other (meta) data is understood when the value given to a metadata property is taken from other semantic resources or databases, e.g., the value http://lexvo.org/id/iso639-3/eng given to the property `omv:hasNaturalLanguage` to indicate an ontology contains English labels. In both cases (ontology and
metadata), we will say that a reference is 'qualified' if: (i) the relationship is explicitly specified (e.g., property coming from a semantic resource) and (ii) the referenced object is unambiguously identified by a GUPRI to be explicitly reused by future users of the ontology.

I3 can be assessed by evaluating the proportional quantity and quality of references to other data and by verifying the resolution of the referenced objects. As in I2, it is up to a community to define the standard databases or resources that ontologies and semantic resources must refer to. Some ontologies may have been explicitly created to index or annotate a specific data resource. I3 is extremely hard to assess as the MOD metadata model does not currently offer any mechanism to inform about the existence of qualified references from ontologies to other data resources. One needs to look explicitly at the content of the ontology. However, we can easily evaluate if the values of certain metadata properties are unambiguous GUPRIs.

<table><thead><tr><td>I3 questions (33 credits)</td><td>Qualified references</td></tr></thead><tbody><tr><td>Q1. Does the ontology provide qualified cross-references to external resources/databases? <b>20 pts</b></td><td></td></tr><tr><td>Q2. If yes, are those cross-references well represented and to unambiguous entities? <b>6 pts</b></td><td></td></tr><tr><td>Q3. Does the ontology use valid URIs to encode some metadata values? <b>7 pts</b></td><td></td></tr><tr><td colspan="2"><b>Related recommendations:</b> (i) FAIRsFAIR P-Rec 10,12,15; (ii) Poveda et al. Rec 7; (iii) 5-stars V principle 3.</td></tr></tbody></table>

Examples:

* Many ontologies such as Gene Ontology, CHEBI or Protein Ontology include qualified cross-references to databases encoded with the `oboInOwl:hasDbXref` property. However, as discussed in Laadhar et al. (2020b), among one million cross-references in thirty OBO Foundry ontologies only 58% were resolvable to an entity. Plus, these entities varied from other ontology concepts (mappings) to various types of cross-references to database, database elements, or even curators. This study showed that better guidelines are indeed necessary to encode cross-references within ontologies.

* Within AgroPortal, the ANAEE Thesaurus is assigned unambiguous URIs for multiple metadata properties, e.g., license (URI provided by CC vocabulary), syntax (URI provided by W3C), subject (URIs provided by AGROVOC), formality level (URI provided by OMV vocabulary), etc.

R1. Ontologies and ontology metadata are richly described with a plurality of accurate and relevant attributes: For metadata, R1 criterion is mainly related to F2 and all the other criteria where metadata requirements are elicited, including R1's sub-criteria, which focus on the context under which the ontologies were generated (e.g., provenance information) or can be reused (e.g., license information). Actually, there is no clear recommendation in FAIRsFAIR nor in Poveda-Villalón et al. about R1. Thus, we concentrate here on what it means for an ontology to be richly described: When an ontology is developed

collaboratively, the metadata of ontology objects (classes, instances, properties) – which are not necessarily the exact same metadata as the ontology itself – should be richly described to trace the provenance of their definition/inclusion in the ontology. Objects metadata typically contains information about what it is, who created an object, when, in which version of the ontology it was created, and other information about its provenance. Sometimes, it is relevant to clarify globally in the ontology metadata which annotation properties are systematically used to capture a certain information in each ontology object. For instance, there are several ways of representing synonym labels (such as the OBO format synonym properties, or the SKOS label properties). Thus, it is a good practice to inform globally at the ontology metadata level which properties are used for this purpose when describing the ontology objects.

R1 can be assessed then by evaluating if objects of an ontology are well described (i.e., if the properties to describe objects are informed and significantly populated). The MOD metadata model offers several properties to describe which metadata properties are used for describing objects in the ontology: `mod:prefLabelProperty`, `mod:synonymProperty`, `mod:definitionProperty`, `mod:authorProperty` and `mod:obsoleteProperty` can be used respectively to specify preferred names, synonyms, definition, author of a class and obsolete status. The definition of an object can be either textual (e.g., using the property declared in `mod:definitionProperty`) and/or logical using OWL constructs such as restrictions (e.g., OWL quantifier, cardinality or has value restrictions) or equivalent classes (e.g., an OWL named class). Moreover, MOD also includes three properties to describe the subclass (or broader/narrower) hierarchy of the semantic resource (`mod:hierarchyProperty`, `mod:obsoleteParent` and `mod:maxDepth`) to be used respectively to inform the hierarchy, the root of an obsolete branch in the ontology and the maximum depth of the hierarchy tree. In our methodology, we will use all the above-listed metadata properties to evaluate the richness of the ontology description and assess R1; of course, other properties could be used such as author and date to better capture ontology object provenance; we would need them in subsequent revision of MOD.

<table><thead><tr><td>R1 questions (32 credits)</td><td>Rich descriptions</td></tr></thead><tbody><tr><td>Q1. Does the ontology provide metadata information about how classes or concepts are defined? <b>5 pts</b></td><td></td></tr><tr><td>Q2. Does the ontology provide metadata information about its hierarchy? <b>3 pts</b></td><td></td></tr><tr><td>Q3. How much of the ontology objects are described with labels? <b>7 pts</b></td><td></td></tr><tr><td>Q4. How much of the ontology objects are defined using a text description? <b>6 pts</b></td><td></td></tr><tr><td>Q5. How much of the ontology objects are defined using a property restriction or an equivalent class? <b>6 pts</b></td><td></td></tr><tr><td>Q6. How much of the ontology objects provide provenance information with annotation properties? <b>5 pts</b></td><td></td></tr><tr><td colspan="2"><b>Related recommendations:</b> none.</td></tr></tbody></table>
Example:

* The Wheat Ontology (CO_321) in AgroPortal (http://agroportal.lirmm.fr/ontologies/CO_321) defines almost all R1 properties. It declares using skos:prefLabel as preferred label property, skos:altLabel as synonym property, rdfs:comment as definition property, and dc:creator as object author property. CO_321 does not declare a property used to tag obsolete objects.

**R1.1. Ontologies and ontology metadata are released with a clear and accessible usage license:** R1.1 criterion is about licensing. Without being a requirement to be FAIR, it is obvious that making ontologies and ontology metadata openly and freely available facilitate reuse. Whatever type of license is being chosen, R1.1 requires a machine-readable declaration and representation of the license. The absence of an explicit license might prevent others from re-using the ontology, even if the ontology is originally intended to be shared and open. FAIRsFAIR authors encourage using open licenses, preferably using Creative Commons 4.0 licensing and Poveda-Villalón et al. recommend using the metadata properties **dct:license**. We note that the site RDF license (https://rdlicense.linkedata.es) provides URIs and machine-understandable RDF descriptions for licenses. In addition, the Creative Commons vocabulary provides several properties for access rights and license descriptions.

R1.1 can be assessed by checking if the license and access rights information are provided and resolvable (especially the license). The MOD metadata model suggests the property **dct:license** for describing license information, and **dct:accessRights** for detailing access rights (who has access to what and to do what). In addition, it proposes properties to cover information on permissions and guidelines to use the ontology (**cc:morePermissions**, **cc:useGuidelines**) as well as the copyright holder (**dct:rightsHolder**). In our methodology, we assume ontology and its metadata are ruled by the same license – situation by default when the metadata are actually described in the same file – but of course, if not, two licenses need to be specified, and the assessment credits shared between ontology and ontology metadata.

**R1.1 questions (37 credits)**

**Licensing & rights**

**Q1.** Is the ontology license clearly specified, with an URI that is resolvable and supports content negotiation? **15 pts**

**Q2.** Are the ontology access rights specified and permissions documented? **7 pts**

**Q3.** Are the ontology usage guidelines and copyright holder documented? **15 pts**

**Related recommendations:** (i) FAIRsFAIR P-Rec 3 and 16 (ii) Poveda et al. Rec 6.

Example:

* AEFO in AgroPortal is explicitly described with a CC-BY4.0 license (https://creativecommons.org/

licenses/by/4.0) and more permissions details are provided in the French Etalab Open License description.³⁶

**R1.2. Ontologies and ontology metadata are associated with detailed provenance:** R1.2 criterion allows to describe all the ontology provenance not already 'covered' by other FAIR principles and to 'make the ontology readable for reuse purposes' as mentioned in FAIRsFAIR P-Rec 13. As any other data, ontology provenance might be described with cross-domain metadata vocabularies – such as Dublin Core, the Provenance Ontology (PROV-O) or the Provenance, Authoring and Versioning (PAV) vocabulary – and as much as possible in a machine-readable form as it is clearly expressed in FAIRsFAIR P-Rec 17. The provenance is also informed by the correct versioning (and archiving of previous versions) of the ontologies through time and by informing which other semantic resources influenced the creation of an ontology – aspect covered by I2. Besides, ontology provenance must also contain information related to the methodology and tools used to build the ontology. As well as rational information on why the ontology was built (e.g., competency questions). Provenance may be understood broadly and cover most of the metadata descriptions of an ontology. Therefore, we focus here on some specific provenance information not already assessed as 'rich metadata' in F2 or in another sub-principle. The use of standard metadata vocabularies to describe metadata/provenance is already evaluated in I2.

R1.2 can be assessed by evaluating the quantity of provenance information provided by the ontology. For this, the MOD metadata model provides several properties – often taken from PROV-O or DCTerms – for describing provenance information (**dct:source**, **prov:wasGeneratedBy**, **prov:wasInvalidatedBy**) and specify how/when objects are added to the ontology (**dct:accrualMethod**, **dct:accrualPeriodicity** and **dct:accrualPolicy**). In addition, MOD regroups several metadata properties for versioning: **owl:versionIRI** (to store a version specific URI, evaluated in F1), **owl:versionInfo** (to store the version information), **dct:hasVersion** to refer to the ontology version, and **omv:hasPriorVersion** to refer to previous versions and **vann:changes** (to document the changes between versions). To document the methodology and tools used to build the ontology, the MOD metadata model offers the properties: **omv:usedOntologyEngineeringTool**, **omv:usedOntologyEngineeringMethodology** and **omv:conformsToKnowledgeRepresentationParadigm**. To document the actors involved in the development of the ontology MOD suggest using the properties: **dct:creator**, **dct:contributor**, **pav:curatedBy**, **schema:translator**. And finally, to document the rationale about why the ontology was built one can use the properties: **omv:designedForOntologyTask**, **mod:competencyQuestion** but also **foaf:fundedBy** to inform which organisation supported the work.
<table><thead><tr><td>R1.2 assessment questions (38 credits)</td><td>Provenance</td></tr></thead><tbody><tr><td>Q1. Does the ontology provide information about the actors involved in its development? 8 pts</td><td></td></tr><tr><td>Q2. Does the ontology provide information about its general provenance? 6 pts</td><td></td></tr><tr><td>Q3. Are the accrual methods and policy of the ontology documented? 6 pts</td><td></td></tr><tr><td>Q4. Is the ontology clearly versioned with version information and links to previous versions? 4 pts</td><td></td></tr><tr><td>Q5. Are the ontology latest changes documented? 2 pts</td><td></td></tr><tr><td>Q6. Are the methodology and tools used to build the ontology documented? 6 pts</td><td></td></tr><tr><td>Q7. Is the ontology rationale documented? 4 pts</td><td></td></tr><tr><td>Q8. Does the ontology inform about its funding organisation? 2 pts</td><td></td></tr><tr><td>Related recommendations: (i) FAIRsFAIR P-Rec 3, 13, 15 and 17; (ii) Poveda et al. Rec 6.</td><td></td></tr></tbody></table>

Example:

* The AGROVOC thesaurus in AgroPortal (http://agroportal.lirmm.fr/ontologies/AGROVOC) provides clear information about the accrual method (www.fao.org/agrovoc/maintenance), policy and periodicity (www.fao.org/agrovoc/how-add-term). It does also describe several other provenance information such as authors, contributors. The ontology engineering tool used is VocBench (http://vocbench.uniroma2.it).

R1.3. Ontologies and ontology metadata meet domain-relevant community standards: R1.3 criterion is one or the more ambiguous and harder to assess as it relates to domain-specific community practices and standards. Ontology design and engineering have been discussed by the scientific community for 30 years, and there are now multiple recognised practices and standard ways of representing things inside an ontology or any kind of semantic resource. With the emergence and specification of the Semantic Web, we now have a set of standard technologies and languages (often recognised as W3C Recommendations) available to build any kind of semantic resources along the semantic continuum: RDFS, SKOS and OWL. In addition, several different communities have produced patterns or guidelines; see for instance respectively the Ontology Design Patterns (Gangemi, 2005) and the OBO Foundry principles (Smith et al., 2007). When building an ontology, even if it is not mandatory because of the diversity of knowledge to capture, it is recommended to follow as much as possible, community or shared principles. Guidelines such as MIRO or metadata profiles such as MOD provide information on what and how to document when reporting an ontology.

Assessing if an ontology follows community standards is hard because one needs to look deeper inside an ontology and evaluate how it is built in relation to state-of-the-art guidelines and practices common in a community. Plus, this is the only FAIR principle that is 'domain specific' i.e., for which a generic assessment measure cannot be implemented. In the O'FAIR methodology, we decided to simply recognise and acknowledge the use of a standard representation language or syntax (points evaluated in II); the recognition, use or endorsement of an ontology by a project or an organisation;

and the inclusion of an ontology in certain curated/guided ontology frameworks or groups such as for instances:

* The OBO Foundry (OBO-F) initiative (Smith et al., 2007) for biological/biomedical ontologies,

* The Planteome ontologies (Cooper et al., 2018) Cliquez ou appuyez ici pour entrer du texte. for reference ontologies for plant,

* The FAO produced vocabularies for agri-food,³⁷

* INRAE curated vocabularies / semantic resources in agriculture and environment,³⁸

* The Crop Ontology project for plant specific trait ontologies (Shrestha et al., 2012),

* The ontologies endorsed by the Wheat Data Initiative (Yeumo et al., 2017),

* The WHO produced vocabularies for health,

* The Industry Foundry Ontologies project.

In R1.3, we will be evaluating if an ontology is included in a group of ontologies respecting certain community practices. By doing so, we delegate to a group/organisation framwork the evaluation process and only rely on their decision. In an open science vision, we may also be inclined, to recognise a higher level of FAIRness for ontologies that are fully open and accessible (even if a restricted private ontology can be FAIR too). For all these, the MOD metadata model suggests the properties mod:ontologyInUse, omv:endorsedBy for describing the information about the projects or organisation endorsing an ontology, mod:group for describing the inclusion of an ontology to a recognised group of ontologies and the property dct:accessRights informs about the free and open availability of an ontology.

<table><thead><tr><td>R1.3 questions (36 credits)</td><td>Community standards</td></tr></thead><tbody><tr><td>Q1. Does the ontology provide information about projects using or organisations endorsing? 10 pts</td><td></td></tr><tr><td>Q2. Is the ontology included in a specific community set or group? (*) 20 pts</td><td></td></tr><tr><td>Q3. Is the ontology openly and freely available? 6 pts</td><td></td></tr><tr><td>(*) This question is community specific.</td><td></td></tr><tr><td>Related recommendations: (i) FAIRsFAIR P-Rec 3, 12, 13; (ii) Poveda et al. Rec 6 and 10.</td><td></td></tr></tbody></table>

Q2 is community-specific and requires expert knowledge to decide whether an ontology belongs to a group and how much a group sets/corresponds to the 'community standards.' In the case of AgroPortal, groups associate ontologies from the same project or organisation (http://data.agroportal.lirmm.fr/groups). Currently, AgroPortal considers 8 groups relevant in the agri-food domain, e.g., OBO-F, Crop Ontology project, INRAE curated and the Wheat Data Initiative ontologies. In this context, we will assign Q2 points as follows: Is the ontology included in the OBO library? 15 pts. Is the part of the main OBO Foundry ontologies? 20 pts Is the ontology included in the WHEAT, CROP or INRAE group? 10 pts
Example:

* The Plant Ontology (PO, http://purl.obolibrary.org/obo/po.owl) is considered as a reference ontology in the community; it is included in the OBO-F group; the WHEAT and CROP group.

# 4 Results

## 4.1 O'FAIRe prototype in AgroPortal

In this section, we describe and demonstrate the O'FAIRe prototype (v2.0) implemented with and within the AgroPortal ontology repository. We implemented the ontology FAIRness assessment methodology previously presented into a web service which executes tests automatically evaluating how a semantic resource stored within AgroPortal responds to the questions. Formally speaking, we use AgroPortal's metadata record to evaluate the level of FAIRness of the corresponding semantic resource. Consequently, we do not evaluate the level of FAIRness of an ontology but the level of FAIRness of the ontology stored within AgroPortal. This distinction is important as several FAIR sub-principles are linked to the repository in which the ontology is hosted. Furthermore, the link between an ontology and its metadata is explicitly represented in AgroPortal (F3.Q2 and Q3 are verified).

The FAIRness assessment questions have been as much as possible implemented in a Java Servlet application. The application is open-source and available for reuse/customisation on GitHub.³⁹ Over O'FAIREe questions: 45 are dependent of the ontology and 16 are determined simply by the fact that the ontology is stored in AgroPortal; which means the repository automatically gives 93 points to an ontology (19% of the total points). Currently, the prototype implements 50/61 questions (82%). The rest of the questions are not yet implemented because we do not have: (i) either a metadata property to store the information necessary to assess the question (e.g., F3.Q1., I2.Q3, I2.Q5, I3.Q1, I3.Q2, R1.Q3, R1.Q4, R1.Q5, R1.Q6 and R1.2.Q5) or (ii) implemented a mechanism to deal with and analyse the ontology content (e.g., F4.Q3, I2.Q2, I2.Q5, I3.Q1). This means that the maximum score an ontology can currently obtain in AgroPortal is 387/478 (normalised score of 81/100).

For the 45 questions⁴⁰ depending on the ontology, O'FAIRe consumes as entry the JSON-LD ontology descriptions as returned by AgroPortal's web service API. For instance, the following call returns the description for the latest version of the Agronomy Ontology (AGRO):⁴¹

http://data.agroportal.lirmm.fr/ontologies/AGRO/latest_submission?display=all

O'FAIRe can assess any semantic resource stored in AgroPortal independently or in batch. When several semantic resources are given as input, O'FAIRe returns the average, min, max and median scores of all the requested resources. The web service takes as input an ontology acronym (local identifier in AgroPortal) or a list of ontology acronyms and returns a JSON output which contains the FAIR scores obtained for each question aggregated by sub-principles and

principles as well as the global score for a resource (i.e., totalScore and normalizedScore). Every score obtained is justified by a short sentence (i.e., explanation), and the list of metadata properties used for the evaluation is given (i.e., properties) so that the user may be aware of how this score was obtained. An example of a web service call for AGRO can be accessed hereafter. The score obtained as of the time of writing is 342 (normalised to 71):

http://services.agroportal.lirmm.fr/ofairer?ontologies=AGRO

The combined parameter allows us to obtain the average score of the group of ontologies given as input. For example, the following call returns the combined FAIR scores for the three related ontologies Animal Trait Ontology for Livestock (ATOL), Environment Ontology for Livestock (EOL) and Animal Health Ontology for Livestock (AHOL):

http://services.agroportal.lirmm.fr/ofairer?ontologies=ATOL,EOL,AHOL&combined

The complete documentation of the prototype is available on GitHub.

The web service was developed to be compliant with any OntoPortal repository assuming they support the MOD 1.4 metadata model. The *url* parameter can be optionally used to specify in which repository (other than AgroPortal) an ontology may be found. For instance, the following call will evaluate the level of FAIRness of AGRO stored in the NCBO BioPortal:⁴²

http://services.agroportal.lirmm.fr/ofairer?url=https://data.bioontology.org&ontologies=AGRO

However, because BioPortal does not implement the equivalent of the MOD model to describe ontology metadata, many properties required to assess FAIRness are not available. Therefore, the score obtained is significantly lower: 182. We are currently discussing within the OntoPortal Alliance (Graybeal et al., 2019) the extension of the metadata model for all the ontology repositories based on the OntoPortal technology.

Equipped with O'FAIRe we have revisited or developed new user interfaces within AgroPortal to display FAIR scores. For instance, it is now possible to order all the semantic resources by FAIR score on the 'Browse' page, which lists all the semantic resources in AgroPortal. As illustrated in Figure 4, a new graphic component allows one to visualise the FAIR scores obtained (here for a group of three ontologies): (left) the *FAIRness wheel* shows the obtained scores over the 15 FAIR sub-principles; (right) the *bar chart* details for each FAIR principle: the total score obtained (i.e., green part) as well as non-obtained points (yellow part) and credits that cannot yet be assigned (grey part) per limits of current implementation. Other interfaces (e.g., the *Summary page*) provides details about an ontology score, metadata properties used and explanations.
<figure><img src="image_4.png" /><figcaption>**Figure 4** O'FAIRe results in AgroPortal for the ATOL/EOL/AHOL. Presented as a FAIRness wheel (left) and bar charts (right)</figcaption></figure>

## 4.2 Synthesised FAIRness analysis for semantic resources in AgroPortal

Thanks to the O'FAIRe prototype, we were able to evaluate the level of FAIRness of semantic resources stored in AgroPortal. The following discussed FAIR scores are strongly linked to the richness and quality of the metadata descriptions in AgroPortal. It is important to recall that within AgroPortal, the metadata are either entered by the ontology developers or by the AgroPortal team, which does some metadata curation/completion. Therefore, the results presented are the scores currently obtained by ontologies in AgroPortal. We expect scores will change in the future with more curation, more tests to cover all the proposed questions, and more awareness of the FAIR score produced by O'FAIRe. Our study is divided into two parts: first, we provide some statistical analysis and explore which FAIR principles are mostly 'respected' and which ones are more problematic. Second, we analyse community-specific outcomes for several groups of ontologies in AgroPortal.

We reviewed 136 public and 13 private ontologies / semantic resources hosted in AgroPortal in January 2022. The corpus includes 113 OWL ontologies, 21 SKOS thesauri, 13 OBO ontologies and 2 terminologies from UMLS. Some of these resources are included into groups cited previously: 37 in the CROP group; 31 in the INRAE group; 24 in the OBO-F group; and 20 in the WHEAT group.

Figure 5 presents normalised FAIR scores for AgroPortal ontologies with a minimum of 25 (this score is obtained by a private ontology that is still under revision by its ontology developers) and a maximum of 70. The average FAIRscore

is 59. Depending on a set threshold (as discussed in Garijo and Poveda-Villalón (2020)), we can say that almost all resources are FAIR if *FAIRScore* needs to be above 40 but only a few are if *FAIRScore* needs to be above 65. This analysis clearly shows:

* FAIR scores are relatively high (median and average around 60) which confirms that ontologies and semantic resources are digital objects that are relatively FAIR 'by design' mostly because they rely on standards and technologies that obviously facilitate FAIRness. The ontology repository has also a significant role in these high scores, which confirms that depositing any kind of digital objects in data repositories make it more FAIR.

* FAIR scores distribution is relatively homogeneous with 80% of the ontologies between 50 and 65. This is mainly because the semantic resources are curated and hosted in an ontology repository and are not published in the wild, so they follow specific metadata editing and curation policies. However, we notice that the 149 semantic resources still have to be improved to fully comply with the 15 FAIR principles. Currently, the highest score is 70 on a maximum score obtainable in AgroPortal of 81. FAIR scores will unavoidably augment after implementing the missing questions in O'FAIRe; however, changes will also be required on the ontology side to reach a normalised score of 100.

Figure 6 and the following paragraphs detail and discuss the average *FAIRPrincipleScores* calculated by the current version of O'FAIRe.
<figure><img src="image_5.png" /><figcaption>**Figure 6** Average normalised FAIRPrincipleScores over 149 ontologies in AgroPortal. Maximum scores obtainable within AgroPortal are mentioned above each bar</figcaption></figure>

<figure><img src="image_6.png" /></figure>

We observe an average normalised score of 70 for F, detailed for each sub-principle in Figure 7. When assessing F1, we observed:

* Good use of URIs as local/primary identifiers, indeed 96 ontologies (65%) provide a valid URI (Q1).

* 65 ontologies (44%) provide an external/secondary identifier which is a DOI in 10 cases (16%). Only 7 external identifiers (10%) are not resolvable, and all DOIs are resolvable (Q2).

* The repository makes explicit the relationship between the ontology and its metadata (Q3); this aspect does not discriminate ontologies if taken from AgroPortal. This is also the case in assessing F3.

* Only 21 ontologies (14%) provide a version specific URI for which 15 are resolvable (Q4).

When assessing F2, we observed almost all ontologies are described with some 'rich' metadata properties: 75% of ontologies are described with at least 8 'MIRO must' metadata properties which are required to obtain Q1 points; 31% of ontologies are described with at least 6 'MIRO

should and optional' metadata properties which are required to obtain Q2 points and 98% of ontologies are described with at least 7 metadata properties with no MIRO correspondence which are required to obtain Q3 points.

When assessing F3, we have assigned 0 points to Q1, but 11 points to Q2 and 10 points to Q3 as all ontologies have their metadata described in an external file (AgroPortal's metadata record) and because AgroPortal offers a bidirectional link between an ontology and its metadata.⁴³ An implementation of O'FAIRe outside of AgroPortal would have required a metadata property to identify and locate explicitly the metadata.

When assessing F4, we observed:

* 51 ontologies (34%) are registered at least in two libraries, mostly the OBO Foundry, Agrisemantics Map of Standards and FAIRSharing; 46 ontologies (30%) are registered in only one library (Q1).

* Only 2 ontologies (1%) are declared available in at least five open ontology repositories; 24 ontologies (16%) are registered at least in four repositories; 11 (7%) in at least in three; 36 (24%) in at least two; and finally, 76 (51%) are registered in only one repository (Q2).
<figure><img src="image_7.png" /><figcaption>**Figure 7** Average normalised *FAIRSubPrincipleScore* for F</figcaption></figure>

We observe an average normalised score of 90 for A, detailed for each sub-principle in Figure 8. When assessing A1, A1.1 and A1.2 we observed:

* 70 ontologies (46%) have at least two resolvable identifiers (A1.Q1). A1.Q2 is directly addressed by the repository which provides an external metadata URI which resolves to the metadata record.

* 58 ontologies (38%) support at least content negotiation with 6 formats; and 86 (57%) support at least 5 formats (A1.Q3). These numbers include 4 formats systematically supported by AgroPortal for metadata.

* 142 ontologies (95%) declare being accessible through a SPARQL endpoint (A1.Q4). Indeed, this field is either provided by the ontology developer or provided by AgroPortal's SPARQL endpoint.

* 100% of ontologies are accessible using an open, free and universal communication protocol: HTTP. This criterion is independent of the ontologies and depends on AgroPortal (Q1, Q2 and Q3 in A1.1).

* Similarly, 100% of ontologies are accessible via authentication and authorisation as this feature is supported by AgroPortal (A1.2).

When assessing A2, we observed all ontologies are totally compliant with the A2 principle because AgroPortal supports versioning (Q1, Q2 and Q3). Plus, 144 ontologies (96%) clearly inform about their status.

We observe an average normalised score of 42 for I, detailed for each sub-principle in Figure 9. When assessing I1, we observed:

* Only 4 ontologies (2%) have not explicitly declared their format (Q1 and Q2).

* 126 ontologies (85%) inform about their syntax (Q3). 130 (90%) describe also the formality level (Q4). These two relatively high scores are due to the fact that the corresponding properties are closely curated by AgroPortal.

* In contrast, only 2 ontologies (1%) inform about their availability in other formats (Q5).

As depicted in Figure 9, I2 and I3 are hard to meet as the scores are impacted by the questions not yet implemented in O'FAIRe (i.e., I2.Q3, I2.Q5, I3.Q1, I3.Q2). When assessing I2, we observed:

* 149 ontologies (100%) import other FAIR vocabularies (I2.Q1), reuse terms for other vocabularies (I2.Q2) and use valid URIs to encode some metadata values (I3.Q3).

<figure><img src="image_8.png" /><figcaption>**Figure 8** Average normalised *FAIRSubPrincipleScore* for A</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>**Figure 9** Average normalised FAIRSubPrincipleScore for I</figcaption></figure>

* 38 ontologies (25%) are aligned to other vocabularies (I2.Q4).

* 120 ontologies (80%) provide information about the relation to or influence of other vocabularies (I2.Q6).

In I3, we focused on the assessment of the included references to other (meta-)data-based without dealing with the qualification aspect; in our case, a 'qualified reference' meant an additional link that would help in understanding the content or the provenance of any ontologies; here we consider projects and endorsing organisations.

Finally, we observe an average normalised score of 38 for R, detailed for each sub-principle in Figure 10. When assessing R1, we observe:

* 114 ontologies (76%) provide at least 4 information about how classes or concepts are defined (Q1) and 133 ontologies (89%) provide at least one information about their hierarchies (Q2). These relatively high scores are explained by the fact that 8 properties are enforced automatically by AgroPortal which uses them in its internal model. If the information is not entered by the ontology developer, one is assigned by default; however, we have observed a better curation of these properties is required as default properties are not necessarily valid. When assessing R1.1, we observe:

* 91 ontologies (61%) provide clear licensing information with a resolvable license URI with content negotiation (Q1). Although this field is frequently included in ontologies or in their documentations, the AgroPortal team works on harmonizing reference to licenses usually using URIs from the CC vocabulary.

* AgroPortal systematically distinguishes between two cases: public or private ontologies. Therefore, in our prototype, we consider 100% ontologies document their access rights thanks to AgroPortal (Q2).

* 140 ontologies (93%) documents at least one information about usage guidelines and copyrights holder (Q3).

When assessing R1.2, we observe:

* 115 ontologies (77%) provide at least one information about the actors (creator, contributor, curator, translator) involved in the development (Q1). And 137 (87%) provide at least one information about general provenance (Q2).

* In contrast, only 6 ontologies (4%) provide information about accrual methods and policies (Q3).

* A good versioning for 137 ontologies (91%) with the description of their current version and links between their previous versions (Q4). Those aspects are also enforced by the repository.

<figure><img src="image_10.png" /><figcaption>**Figure 10** Average normalised FAIRSubPrincipleScore for R</figcaption></figure>
* 76 ontologies (51%) inform about the tools and methods used in the development (Q6); 66 ontologies (44%) inform about their funding organisation (Q8). But, only 10 ontologies (6%) describe their rationale e.g., competency questions and related tasks (Q7).

When assessing R1.3, we observe:

* A good description of endorsing organisations or related projects in 123 ontologies (82%) information are available (Q1).

* 136 ontologies (92%) are open (Q3) among them 92 ontologies (61%) are included in a specific community set or group (Q2).

Figure 11 describes the average normalised FAIR scores for each studied group of ontologies in AgroPortal namely, INRAE, OBO-F, CROP and WHEAT and Table 2 shows statistics. Hereafter, we highlight some findings:

* Scores (see Figure 11) are similar for all sub-principles except in F1, F2, F4, A1, R1.1 and R1.3.

* Average FAIR scores are relatively the same (see Table 2). It is clear that fulfilling the FAIR principles is challenging for all communities, even the OBO-F that has been doing a considerable effort in ontology and metadata maintenance (Jackson et al., 2021).

* Ontologies in the OBO-F group have the higher F1 scores (identifiers) thanks to a rigorous use of PURLs; however, the CROP group has a low score because 99% of ontologies are without URIs.

* Ontologies in OBO-F group adopt a relatively 'good' usage of version URIs compared to other groups; 65% ontologies have a resolvable version URIs but only 40% in WHEAT, 1% in INRAE and 0% in CROP.

* Ontologies in the WHEAT group follow 'good' practices in reusability in particular, aspects related to license, access rights and funding organisation.

* All ontologies in the CROP group have resolvable licenses. We notice also that INRAE scores are also good because 27 ontologies (84%) are fully compliant with R1.1.Q1 (license). This reflects efforts from these communities to clarify licensing.

* 72% of CROP ontologies describe their funding organisation (R1.2 Q8), but only 58% in INRAE, 40% in WHEAT and 26% in OBO-F.

* The majority of OBO-F ontologies (99%) do not document well the permissions, usage guidelines and copyrights holder (R1.1.Q3).

Best ontology in INRAE is Agriculture Experiments Ontology (AEO), in CROP is Banana Ontology (CO_325) and in WHEAT is Environment Ontology.

<table><caption>**Table 2** FAIR scores statistics for the 4 studied groups</caption><thead><tr><th></th><th>INRAE<br>n=31</th><th>OBO-F<br>n=24</th><th>CROP<br>n=37</th><th>WHEAT<br>n=20</th></tr></thead><tbody><tr><th>Min</th><td>53</td><td>51</td><td>25</td><td>44</td></tr><tr><th>Max</th><td>67</td><td>71</td><td>67</td><td>70</td></tr><tr><th>Average</th><td>61</td><td>65</td><td>59</td><td>63</td></tr><tr><th>Median</th><td>61</td><td>67</td><td>60</td><td>63</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>(a) INRAE FAIRness</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) OBO-F FAIRness</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c) CROP FAIRness</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>(d) WHEAT FAIRness</figcaption></figure>

# 5 Discussion and perspectives

## 5.1 About FAIRness assessment and O'FAIRness

Our projection of the 15 FAIR principles for ontology and ontology metadata shows they are easy to interpret at first sight because they are described with short sentences and a clear terminology. However, digging in more details, we can say only 8 sub-principles are simple to put into practice and evaluate; namely, F1, F3, F4, A1, A1.1, A1.2, I1, R1.1; the rest are not obvious and are subjective to some interpretation:
* F2 deserves a specific treatment as it is about metadata – in general – but many other sub-principles are also about a specific metadata aspect. Possibly, anything describing an ontology that does not explicitly fit in another principle can be used, somehow, to assess F2. In our methodology, the upper bound is set by properties included in MOD1.4.

* F2 and R1 are somehow overlapping, with ‘rich metadata’ in F2 and ‘richly described with a plurality of (...) attributes’ in R1. The difference between them could be clearly specified. We have decided to focus R1 on metadata related to explicitly describing the content of ontologies (labels, definitions, hierarchy, etc.).

* A2 requires clear ontology and metadata preservation strategies; these strategies are not trivial to verify/assess often by lack of historical or versioning information.

* I2 focuses on the use of FAIR vocabularies, which inherently creates a recursive situation when assessing the level of FAIRness of the reused vocabularies; this is not really addressed in our methodology, but a perspective would be to weigh the score of I2 questions with a FAIR Score obtained by our own methodology when available.

* I3 is a little vague or too wide as it relates to all the possible ‘relations’ between ontologies and data; our choice to evaluate it with cross-references is certainly limited.

* R1.2 is very hard to evaluate as provenance can be understood with a very large spectrum. An approach like ours based on metadata descriptions is limited by the availability of metadata properties to describe metadata provenance. MOD1.4 includes 6 explicit provenance properties but others could be considered and included in the methodology.

* R1.3 is community-specific and thus deserves a community-specific-subjective interpretation and evaluation; what is a standard in the biomedical domain might not be relevant in agronomy or industry, etc. Some communities have established their guidelines, others not yet.

## 5.2 About O'FAIRe questions

A key feature of our approach is that the final credits are independent of the number of questions. FAIRness assessment questions are the backbone of O'FAIRe methodology; they can be adapted or completed to refine the evaluation of one specific aspect without changing the overall methodology. For example, if a fourth question related to a sub-principle is added, then the total number of credits assigned for this sub-principle will be re-dispatched on the four questions rather than on three, then refining the assessment process without changing the overall balance of principles relatively to each other. The current list of questions are derived both from our experience working on

ontology repository and services for years (Noy et al., 2009) but also our participation – and sometime leading roles – in interest groups such as EUDAT Semantics, RDA VSSIG, GO-FAIR INs, FAIRsFAIR workshops (Bahim et al., 2020). Many of O'FAIRe questions are generic – i.e., not specific to ontologies – and could be reused by other communities for implementing FAIRness assessment methods for other kinds of digital objects. Of course, there are a few limits in our methodology:

* Owing to a few missing metadata properties in MOD and some limits in AgroPortal the current O'FAIRe prototype does not implement 11 questions. MOD is currently being revised and a v2 – based on the DCAT W3C Recommendation – shall address our needs with respect to missing properties. AgroPortal technical limitations will be addressed along the way to improve the system.⁴⁵

* O'FAIRe approach to rely on metadata properties to evaluate a question means that it is bounded to a metadata model (such as MOD1.4) that must offer the required properties. It is dependent on either: (i) the population of the metadata by the developer or curator; (ii) other components to compute automatically the value of a metadata property such as, the component computing metrics in AgroPortal.

* The integrated FAIRness assessment grid which fixes the importance and balance of each FAIR principles and sub-principles shall be updated to include recent or future relevant works e.g., as proposed by Le Franc et al. (2020), or the final iteration of the FAIRsFAIR recommendations (to come in 2022). Note, eventually the guidelines and recommendations shall certainly converge and new entries will mathematically less influence the grid. Therefore, we do not expect the grid to change much after a certain time.

* O'FAIRe v2's 61 questions would need a community evaluation now that a prototype is available and demonstrable. Our previous tentative discussions of these questions before their implementation turned out unproductive because they were too detached from reality. In the future, it would be useful to collect community feedback to consolidate and validate our proposal and avoid biases that certainly come from our interpretation of how to make semantic resources FAIR. For instance, Annexes 4 and 5 contains subjective coarse-grained appreciations that in the future we would called for vote in the context of RDA VSSIG.

## 5.3 About O'FAIRe implementation and results

With the AgroPortal repository, our results show interoperability and reusability aspects are somehow less addressed. Comparatively, accessibility is quite good, intrinsically because semantic resources are digital objects, mostly relying on semantic web technologies which address (with HTTP and URIs) some of the FAIR accessibility
requirements; in our implementation of O'FAIRe in
AgroPortal's accessibility is also taken care by the repository.

O'FAIRe implementation presented Sub-section 0 differs
from existing initiatives, as it proposes an automatic
FAIRness assessment and offers suggestions for improving
the obtained scores. It is compatible with any ontology
repository based on the OntoPortal technology (originally
developed for the NCBO BioPortal) and implementing the
MOD1.4 ontology metadata model (or an equivalent one
offering the same coverage with metadata properties).
Currently, it can be used on AgroPortal, previously presented,
and the SIFR BioPortal, a French biomedical terminologies
repository (http://bioportal.lirmm.fr) (Jonquet et al., 2016)
built to implement the SIFR Annotator ontology-based text
annotation tool (Tchechmedjiev et al., 2018). Collaborations
within the OntoPortal Alliance will enable us to extend and
maybe customise O'FAIRe for other repositories such as the
NCBO BioPortal, LifeWatch's EcoPortal or Fraunhofer's
MatPortal. Eventually, we can also refactor our tool to
evaluate the level of FAIRness of an ontology simply based
on the ontology file – and its included or referenced metadata.
But we believe that today, such an approach would be
unfruitful due to the lack of metadata descriptions present in
ontologies taken into the wild (Jonquet et al., 2018a).

One limitation of our analysis is that the current
FAIRness assessment scores are calculated on the basis of the
current state of AgroPortal's ontologies descriptions. Indeed,
the scores will necessarily change in the near future with the
release of O'FAIRe in AgroPortal (February 2022) and its
usability for ontology developers to improve the FAIR score
of their resources. AgroPortal team will also necessarily use
this information in the future metadata curation process. This
situation is in fact, desirable as the overall objective of
O'FAIRe is indeed to augment the adoption of the FAIR
principles for ontologies and semantic resources. In other
words, it means that the impact of O'FAIRe will be
measurable in a few months after it has been used to increase
the level of FAIRness.

6 Conclusion

FAIR is a journey, not necessarily a destination; a key
element to become FAIR is to use FAIR vocabularies and
ontologies. But, how do we know if an ontology is FAIR?
The need to construct a common framework to define
ontology FAIRness and harmonise multiple existing
assessment guidelines and methods (generic and specific)
was clearly identified by the RDA Vocabulary and Semantic
Services Interest Group and the H2020 FAIRsFAIR actions.
In this paper, we guide the semantic community to put the
FAIR principles into practice and enable them to qualify the
degree of FAIRness of their semantic resource. Our work
addresses several scientific and technical challenges
regarding the implementation of the 15 FAIR principles for
ontologies and semantic resources. It provides solutions that
would facilitate the adoption of FAIR. Based on preliminary

results, we analysed how easy or hard it is for semantic
resources to adhere to the principles; analysis led us to many
conclusions and perspectives for FAIR ontologies.

Acknowledgements and contributions

This work has been supported by the *Data to Knowledge in Agronomy and Biodiversity* project (D2KAB – www.d2kab.org – ANR-18-CE23-0017) and the project ANR *French participation in GO FAIR Food Systems Implementation Network* (FooSIN – https://foosin.fr – ANR19-DATA-0019). We also thank the VSSIG (Vocabulary and Semantic Services Interest Group) of the Research Data Alliance and the H2020 FAIRsFAIR project T2.2 on ‘FAIR Semantics’ for fruitful discussions and exchanges. Emna Amdouni compiled related work. Emna Amdouni and Clement Jonquet conceived the methodology and jointly wrote the article. Emna Amdouni and Syphax Bouazzouni implemented O’FAIRe prototype and integration in AgroPortal. Clement Jonquet envisioned and led the research action.

References

Amdouni, E. and Jonquet, C. (2021) ‘FAIR or FAIRer? An integrated quantitative FAIRness assessment grid for semantic resources and ontologies’, *Proceedings of the 15th International Conference on Metadata and Semantics Research*, Springer, Madrid, Spain, pp.67–80. Doi: 10.1007/978-3-030-98876-0_6.

Amdouni, E. and Jonquet, C. (2021) ‘FAIR or FAIRer? An integrated quantitative FAIRness assessment grid for semantic resources and ontologies’, *Hall Open Science*. Doi: 10.13140/RG.2.2.23265.84321.

Bahim, C. et al. (2020) ‘The FAIR data maturity model: an approach to harmonise FAIR assessments’, *Data Science Journal*, Vol. 19, No. 1, pp.1–7. Doi: 10.5334/DSJ-2020-041/METRICS/.

Bizer, C., Heath, T. and Berners-Lee, T. (2009) ‘Linked data – the story so far’, *Semantic Web and Information Systems*, Vol. 5, No. 3, pp.1–22. Doi: 10. T.4018/jswis.2009081901.

Caracciolo, C. et al. (2020) ‘39 hints to facilitate the use of semantics for data on agriculture and nutrition’, *Data Science Journal*, Vol. 19, No. 1. Doi: 10.5334/dsj-2020-047.

Clarke, D.J.B. et al. (2019) ‘FAIRshake: toolkit to evaluate the FAIRness of research digital resources’, *Cell Systems*, Vol. 9, No. 5, pp.417–421. Doi: 10.1016/J.CELS.2019.09.011.

Collins, S. et al. (2018) *Turning FAIR into Reality: Final Report and Action Plan from the European Commission Expert Group on FAIR data*. Doi: 10.2777/54599.

Cooper, L. et al. (2018) ‘The Planteome database: an integrated resource for reference ontologies, plant genomics and phenomics’, *Nucleic Acids Research*, Vol. 46, No. D1, pp.D1168–D1180. Doi: 10.1093/nar/gkx1152.

Côté, R.G., Jones, P., Apweiler, R. and Hermjakob, H. (2006) ‘The ontology lookup service, a lightweight cross-platform tool for controlled vocabulary queries.’, *BMC Bioinformatics*, Vol. 7, p.97. Doi: 10.1186/1471-2105-7-97.

Courtot, M. et al. (2011) ‘MIREOT: the minimum information to reference an external ontology term’, *Applied Ontology*, Vol. 6, No. 1, pp.23–33. Doi: 10.3233/AO-2011-0087.
Cox, S. and Yu, J. (2022) ‘OzNome 5-star tool: a rating system for making data FAIR and trustable’, *eResearch Australasia Conference*. Available: https://conference.eresearch.edu.au/2017/08/oznome-5-star-tool-a-rating-system-for-making-data-fair-and-trustable/ (accessed on 31 March 2022).

Coxid, S.J.D., Gonzalez-Beltranid, A.N., Magagna, B. and Marinescu, M-C. (2021) ‘Ten simple rules for making a vocabulary FAIR’, *PLOS Computational Biology*. Doi: 10.1371/journal.pcbi.1009041.

D’Aquin, M. and Noy, N.F. (2012) ‘Where to publish and find ontologies? A survey of ontology libraries’, *Web Semantics*, Vol. 11, pp.96–111. Doi: 10.1016/j.websem.2011.08.005.

David, R. et al. (2020) ‘Fairness literacy: the achilles heel of applying fair principles’, *Data Science Journal*, Vol. 19, No. 1, pp.1–11. Doi: 10.5334/DSJ-2020-032/METRICS/.

Devaraju, A. and Huber, R. (2021) ‘An automated solution for measuring the progress toward FAIR research data’, *Patterns*, Vol. 2, No. 11. Doi: 10.1016/J.PATTER.2021.100370.

Devaraju, A. et al. (2021) ‘From conceptualization to implementation: fair assessment of research data objects’, *Data Science Journal*, Vol. 20, No. 1, pp.1–14. Doi: 10.5334/DSJ-2021-004/METRICS/.

Ding, Y. and Fensel, D. (2001) ‘Ontology library systems: the key to successful ontology re-use’, *Proceedings of the 1st Semantic Web Working Symposium*, pp.93–112.

Dutta, B., Toulet, A., Emonet, V. and Jonquet, C. (2017) ‘New generation metadata vocabulary for ontology description and publication’, *Proceedings of the 11th Metadata and Semantics Research Conference*, Vol. 755. Doi: 10.1007/978-3-319-70863-8_17.

Fiore, N., Magagna, B. and Goldfarb, D. (2017) ‘EcoPortal: a proposition for a semantic repository dedicated to ecology and biodiversity’, *Proceedings of the 2nd International Workshop on Semantics for Biodiversity*, S4BioDiv’17, 2017.

Frasson, C. and Kostopoulos, G. (Eds) (2017) *Brain Function Assessment in Learning*, Springer International Publishing, Cham. Doi: 10.1007/978-3-319-67615-9.

Frey, J., Streitmatter, D., Götz, F., Hellmann, S. and Arndt, N. (2020) ‘DBpedia Archivo: a web-scale interface for ontology archiving under consumer-oriented aspects’, *Proceedings of the International Conference on Semantic Systems*, pp.19–35. Doi: 10.1007/978-3-030-59833-4_2/TABLES/2.

Gangemi, A. (2005) ‘Ontology design patterns for semantic web content’, *Proceedings of the 4th International Semantic Web Conference*, pp.262–276. Doi: 10.1007/11574620_21.

Garijo, D. and Poveda-Villalón, M. (2020) ‘Best practices for implementing FAIR vocabularies and ontologies on the web’, in Cota, G., Daquino, M. and Pozzato, G.L. (Eds): *Applications and Practices in Ontology Design, Extraction, and Reasoning*, IOS Press. Doi: 10.3233/SSW200034.

Garijo, D., Corcho, O. and Poveda-Villalón, M. (2021) *FOOPS!: an ontology pitfall scanner for the FAIR principles*. Available online at: http://ceur-ws.org/Vol-2980/ (accessed on 31 March 2022).

Graybeal, J., Jonquet, C., Fiore, N. and Musen, M.A. (2019) ‘Adoption of BioPortal’s Ontology Registry Software: The Emerging OntoPortal Community’, *Proceedings of the 13th Research Data Alliance Plenary Meeting*, Philadelphia, USA.

Hartmann, J., Palma, R. and Gómez-Pérez, A. (2009) ‘Ontology Repositories’, *Handbook on Ontologies*, pp.551–571. Doi: 10.1007/978-3-540-92673-3.

Hasnain, A. and Rebholz-Schuhmann, D. (2018) ‘Assessing FAIR data principles against the 5-star open data principles’, *The Semantic Web: ESWC 2018 Satellite Events*, pp.469–477. Doi: 10.1007/978-3-319-98192-5_60/TABLES/1.

Jackson, R. et al. (2021) ‘OBO Foundry in 2021: operationalizing open data principles to evaluate ontologies’, *Database*, pp.1–9. Doi: 10.1093/DATABASE/BAAB069.

Jackson, R.C., Balhoff, J.P., Douglass, E., Harris, N.L., Mungall, C.J. and Overton, J.A. (2019) ‘ROBOT: a tool for automating ontology workflows’, *BMC Bioinformatics*, Vol. 20, No. 1, pp.1–10. Doi: 10.1186/S12859-019-3002-3/FIGURES/1.

Janowicz, K., Hitzler, P., Adams, B., Kolas, D. and Vardeman, C. (2014) ‘Five stars of linked data vocabulary use’, *Semantic Web*, Vol. 5, No. 3, pp.173–176. Doi: 10.3233/SW-140135.

Jonquet, C. (2018) ‘FAIR data requires FAIR ontologies, how do we do?’, *F1000Res*, Vol. 7. Doi: 10.7490/F1000RESEARCH.1115343.1.

Jonquet, C. (2022) *Ontology repository and ontology-based services – challenges, contributions and applications to biomedicine & agronomy*. Available online at: https://tel.archives-ouvertes.fr/

Jonquet, C. et al. (2018b) ‘AgroPortal: a vocabulary and ontology repository for agronomy’, *Computers and Electronics in Agriculture*, Vol. 144, pp.126–143. Doi: 10.1016/j.compag.2017.10.012.

Jonquet, C., Annane, A., Bouarech, K., Emonet, V. and Melzi, S. (2016) ‘SIFR BioPortal: Un portail ouvert et générique d'ontologies et de terminologies biomédicales françaises au service de l'annotation sémantique’, *Journées Francophones d'Informatique Médicale*, Genève, Suisse.

Jonquet, C., Toulet, A., Dutta, B. and Emonet, V. (2018a) ‘Harnessing the power of unified metadata in an ontology repository: the case of AgroPortal’, *Data Semantics*, Vol. 7, pp.191–221. Doi: 10.1007/s13740-018-0091-5.

Laadhar, A., Abrahão, E. and Jonquet, C. (2020a) ‘Analysis of term reuse, term overlap and extracted mappings across AgroPortal semantic resources’, *Proceedings of the 22nd International Conference on Knowledge Engineering and Knowledge Management*, pp.71–87. Doi: 10.1007/978-3-030-61244-3_5/FIGURES/3.

Laadhar, A., Abrahão, É. and Jonquet, C. (2020b) ‘Investigating one million XRefs in thirty ontologies from the OBO world’, *Proceedings of the 11th International Conference on Biomedical Ontologies*, pp.1–12. Available online at: https://researchr.org/publication/LaadharAJ20-0/bibtex (accessed on 31 March 2022).

Le Franc, Y., Coen, G., Essen, J.P., Bonino, L., Lehväslaiho, H. and Staiger, C. (2020) ‘D2.2 FAIR semantics: first recommendations’, Doi: 10.5281/zenodo.3707985.

Malone, J., Stevens, R., Jupp, S., Hancock, T., Parkinson, H. and Brooksbank, C. (2016) ‘Ten simple rules for selecting a bio-ontology’, *PLOS Computational Biology*, Vol. 12, No. 2, p.6. Doi: 10.1371/journal.pcbi.1004743.

Matentzoglu, N., Malone, J., Mungall, C. and Stevens, R. (2018) ‘MIRO: guidelines for minimum information for the reporting of an ontology’, *Journal of Biomed Semantics*, Vol. 9, No. 1, p.6. Doi: 10.1186/s13326-017-0172-7.

Mons, B. (2018) *Data Stewardship for Open Science*, Chapman and Hall/CRC. Doi: 10.1201/9781315380711.

Mons, B., Neylon, C., Velterop, J. and Dumontier, M. (2017) ‘Cloudy, increasingly FAIR; revisiting the FAIR data guiding principles for the European Open Science Cloud’, *Information Services and Use*, Vol. 37, pp.49–56. Doi: 10.3233/ISU-170824.
Noy, N.F. et al. (2009) ‘BioPortal: ontologies and integrated data resources at the click of a mouse’, *Nucleic Acids Research*, Vol. 37, pp.170–173. Doi: 10.1093/nar/gkp440.

Poveda-Villalón, M., Espinoza-Arias, P., Garijo, D. and Corcho, O. (2020) ‘Coming to Terms with FAIR Ontologies’, *Proceedings of the 22nd International Conference on Knowledge Engineering and Knowledge Management*, pp.255–270. Doi: 10.1007/978-3-030-61244-3_18/TABLES/1.

Rosnet, T., Lefort, V., Devignes, M-D. and Gaignard, A. (2021) ‘FAIR-Checker, a web tool to support the findability and reusability of digital life science resources’, *JOIBM*, Paris, Zenodo. Doi: 10.5281/ZENODO.5914307.

Schultes, E., Magagna, B., Hettne, K.M., Pergl, R., Suchánek, M. and Kuhn, T. (2020) ‘Reusable FAIR implementation profiles as accelerators of FAIR convergence’, *Advances in Conceptual Modeling*, pp.138–147. Doi: 10.1007/978-3-030-65847-2_13/FIGURES/2.

Schwardmann, U. (2020) ‘Digital objects – FAIR digital objects: which services are required?’, *Data Science Journal*, Vol. 19, No. 1, pp.1–6. Doi: 10.5334/DSJ-2020-015/METRICS/.

Shrestha, R. et al. (2012) ‘Bridging the phenotypic and genetic data useful for integrated breeding through a data annotation using the crop ontology developed by the crop communities of practice’, *Frontiers in Physiology*, Vol. 3, pp.1–10. Doi: 10.3389/fphys.2012.00326.

Smith, B. et al. (2007) ‘The OBO foundry: coordinated evolution of ontologies to support biomedical data integration’, *Nature Biotechnology*, Vol. 25, No. 11, pp.1251–1255, Doi: 10.1038/nbt1346.

Suarez-Figueroa, M.C., Hartmann, J., Sure, Y., Haase, P. and Suarez-Figueroa, M. (2005) ‘OMV–Ontology metadata vocabulary’, *Proceedings of the Workshop on Ontology Patterns for the Semantic Web*, pp.906–915.

Tchechmedjiev, A., Abdaoui, A., Emonet, V., Zevio, S. and Jonquet, C. (2018) ‘SIFR annotator: ontology-based semantic annotation of French biomedical text and clinical notes’, *BMC Bioinformatics*, Vol. 19, pp.405–431. Doi: 10.1186/s12859-018-2429-2.

Toulet, A., Dutta, B. and Jonquet, C. (2018) *Assessing the Practice of Ontology Metadata: A Survey Result*, Montpellier, France. Doi: 10.5281/zenodo.3484530.

Vandenbussche, P-Y., Atemezing, G.A., Poveda-Villalon, M. and Vatant, B. (2014) ‘Linked open vocabularies (LOV): a gateway to reusable semantic vocabularies on the web’, *Semantic Web*, Vol. 1, pp.1–5. Doi: 10.3233/SW-160213.

Wilkinson, M.D. et al. (2016) ‘The FAIR guiding principles for scientific data management and stewardship’, *Scientific Data*, Vol. 3. Doi: 10.1038/sdata.2016.18.

Wilkinson, M.D. et al. (2019) ‘Evaluating FAIR maturity through a scalable, automated, community-governed framework’, *Scientific Data*, Vol. 6, No. 1. Doi: 10.1038/S41597-019-0184-5.

Wilkinson, M.D., Sansone, S-A., Schultes, E., Doorn, P., Da Silva Santos, L.O.B. and Dumontier, M. (2018) ‘A design framework and exemplar metrics for FAIRness’, *Scientific Data*, Vol. 5. Doi: 10.1038/sdata.2018.118.

Yeumo, E.D. et al. (2017) ‘Developing data interoperability through standards: a wheat community use case’, *F1000 Research*, Vol. 6. Doi: 10.12688/f1000research.12234.1.

Zeng, M.L. (2008) ‘Knowledge organization systems (KOS)’, *Knowledge Organization*, Vol. 35, Nos. 2/3, pp.160–182.

Zeng, M.L. and Clunis, J. (2020) ‘FAIR + FIT: guiding principles and functional metrics for linked open data (LOD) KOS products’, *Journal of Data and Information Science*, Vol. 5, No. 1, pp.93–118. Doi: 10.2478/JDIS-2020-0008.

## Notes

1. In this paper, we consider the terms ontologies, terminologies, thesaurus and vocabularies as a type of knowledge organization systems (Zeng, 2008) or semantic artifacts (Le Franc et al., 2020) or semantic resources (Caracciolo et al., 2020). Those are the subjects we are interested in making FAIR. For simplicity, we will sometimes use ‘ontology’ as an overarching word.

2. RDA Vocabulary and Semantic Services Interest Group.

3. With regards to the expression used by the GO FAIR community (Schultes et al., 2020), we can say our methodology is a FAIR Implementation Profile (FIP) for semantic resources and it relies on FIP (meta) data related questions.

4. https://peta-pico.github.io/FAIR-nanopubs/principles/index-en.html

5. https://www.fair-access.net.au/fair-statement

6. https://commonfund.nih.gov/commonss

7. www.fairsfair.eu

8. www.go-fair.org/implementation-networks/overview/food-systems

9. https://www.surveymonkey.com/r/fairdat

10. https://certificates.theodi.org/en/about/badgelevels

11. https://5stardata.info

12. https://slideplayer.com/slide/16324780/

13. https://satifyd.dans.knaw.nl

14. https://bigdata.cgiar.org/resources/gardian

15. https://ardc.edu.au/resources/aboutdata/fair-data

16. https://www.f-uji.net

17. https://faircookbook.elixir-europe.org

18. https://www.w3.org/TR/swbp-vocab-pub

19. https://github.com/sifrproject/MOD-Ontology

20. For instance, MOD does not require the use of a specific authorship property but rather encodes that `dc:creator`; `schema:author`, `foaf:maker`, or `pav:createdBy` can be used to say so.

21. https://github.com/FAIRsFAIR-Project/FAIRSemantics/issues

22. https://fairvocabularies.github.io/makeVocabularyFAIR

23. https://archivo.dbpedia.org

24. The Semantic Web community also says ‘dereferenceable’.

25. https://obofoundry.org

26. https://fairdigitalobjectframework.org/

27. http://www.obofoundry.org/faq/how-do-i-edit-metadata.html

28. https://github.com/OBOFoundry/OBOFoundry.github.io/edit/master/ontology/pato.md

29. See http://edamontology.org/EDAM.uris for details about URIs

30. http://agroportal.lirmm.fr/ontologies/AFEO

31. http://agroportal.lirmm.fr/ontologies/ANAEETHES

32. http://data.agroportal.lirmm.fr/ontologies/ANAEETHES/latest_submission?display=all

33. We voluntarily do not use the term ‘format’ as it creates a confusion between the ontology syntax and representation language. We will use the term format when there is no distinction e.g., TXT, CSV or PDF.

34. https://nkos.dublincore.org

35. To avoid confusion, in this section we use ‘ontologies’ for the semantic resources using and ‘vocabularies’ for resources being used.

36. https://www.etalab.gouv.fr/wp-content/uploads/2018/11/open-licence.pdf
37 http://datalab.review.fao.org/datalab/caliper

38 https://vocabulaires-ouverts.inrae.fr

39 https://github.com/agroportal/fairness

40 Except A1.Q3 that is related to both metadata and repository.

41 AgroPortal's internal metadata model implements MOD1.4 but does not always use MOD suggested properties to stay backward compatible with BioPortal. The correspondence between MOD1.4 properties and O'FAIRe used properties in AgroPortal is available.

42 An API Key is required to query the NCBO BioPortal. Therefore, the REST Web service call must be completed by a parameter &apikey='...' as provided by

BioPortal. The constraint exists also on AgroPortal, but in our case the system uses a default API Key to facilitate access. During the reviewing process, one can use the following API key for Ncbo Bioportal: 4a482fad-9126-4696-a35a-7d42c05b82f0

43 In AgroPortal, an 'ontology' is linked to several 'submissions'. Most of the metadata is attached to the submission object.

44 For example, we do not yet have a mechanism to evaluate the quantities (i.e., the ratio over the size of the ontology) and qualities of mappings (I2) or qualified references (I3) from an ontology. As another example, in some cases, O'FAIRe just checks the existence of a value for a metadata property but not the exactitude or relevance of that value.
# ANNEXES

<table><caption>## Annex 1: Integrated FAIRness assessment grid for semantic resources and ontologies</caption><thead><tr><th></th><th>Principle</th><th>Baseline</th><th>SHARC</th><th>FDMM</th><th>5-stars V</th><th>MIRO</th><th>FAIRsFAIR</th><th>Poveda et al.</th><th>Credits</th></tr></thead><tbody><tr><td rowspan="4">F</td><td>F1</td><td>10</td><td>8</td><td>12</td><td>0</td><td>6</td><td>2</td><td>3</td><td>41</td><td rowspan="4">113</td></tr><tr><td>F2</td><td>10</td><td>8</td><td>2</td><td>0</td><td>5</td><td>1</td><td>1</td><td>27</td></tr><tr><td>F3</td><td>10</td><td>8</td><td>2</td><td>0</td><td>0</td><td>1</td><td>0</td><td>21</td></tr><tr><td>F4</td><td>10</td><td>8</td><td>3</td><td>0</td><td>0</td><td>2</td><td>1</td><td>24</td></tr><tr><td rowspan="4">A</td><td>A1</td><td>10</td><td>6</td><td>18</td><td>3</td><td>3</td><td>1</td><td>2</td><td>43</td><td rowspan="4">113</td></tr><tr><td>A1.1</td><td>10</td><td>6</td><td>11</td><td>0</td><td>0</td><td>1</td><td>0</td><td>28</td></tr><tr><td>A1.2</td><td>10</td><td>6</td><td>5</td><td>0</td><td>0</td><td>1</td><td>0</td><td>22</td></tr><tr><td>A2</td><td>10</td><td>6</td><td>3</td><td>0</td><td>0</td><td>1</td><td>0</td><td>20</td></tr><tr><td rowspan="3">I</td><td>I1</td><td>10</td><td>4</td><td>12</td><td>1</td><td>12</td><td>3</td><td>2</td><td>44</td><td rowspan="3">109</td></tr><tr><td>I2</td><td>10</td><td>4</td><td>7</td><td>0</td><td>9</td><td>2</td><td>0</td><td>32</td></tr><tr><td>I3</td><td>10</td><td>4</td><td>12</td><td>1</td><td>3</td><td>2</td><td>1</td><td>33</td></tr><tr><td rowspan="4">R</td><td>R1</td><td>10</td><td>9</td><td>3</td><td>1</td><td>6</td><td>0</td><td>3</td><td>32</td><td rowspan="4">143</td></tr><tr><td>R1.1</td><td>10</td><td>9</td><td>12</td><td>0</td><td>3</td><td>2</td><td>1</td><td>37</td></tr><tr><td>R1.2</td><td>10</td><td>9</td><td>3</td><td>0</td><td>12</td><td>3</td><td>1</td><td>38</td></tr><tr><td>R1.3</td><td>10</td><td>9</td><td>12</td><td>0</td><td>0</td><td>3</td><td>2</td><td>36</td></tr><tr><td colspan="2">Total credits</td><td colspan="2">478</td><td colspan="2"></td><td colspan="2"></td><td colspan="2"></td><td></td></tr></tbody></table>

<table><caption>## Annex 2: List of 71 'extra' metadata properties from MOD 1.4 and their alignment with MIRO guidelines</caption><tr><td>Must</td><td>omv:acronym, dct:title, dct:alternative, skos:hiddenLabel, dct:description, foaf:page, omv:resourceLocator, omv:keywords, dct:coverage, foaf:homepage, vann:example, vann:preferredNamespaceUri, void:uriRegexPattern, idot:exampleIdentifier, dct:publisher, dct:subject, owl:backwardCompatibleWith, door:comesFromTheSameDomain, omv:knownUsage, dct:audience, doap:repository, doap:bugDatabase, doap:mailing-list, mod:hasEvaluation</td></tr><tr><td>Should</td><td>mod:metrics, omv: numberOfClasses, omv: numberOfIndividuals, omv: numberOfProperties, mod: numberOfDataProperties, mod: numberOfObjectProperties, omv: numberOfAxioms, mod: numberOfLabels, mod:byteSize</td></tr><tr><td>Optional</td><td>vann:preferredNamespacePrefix</td></tr><tr><td>No mapping</td><td>dct:language, dct:abstract, mod: analytics, dct: bibliographicCitation, rdfs:comment, foaf: depiction, foaf: logo, voaf: toDoList, schema:award, schema:associatedMedia, owl:incompatibleWith, dct:hasPart, schema:workTranslation, door:hasDisparateModelling, voaf:usedBy, voaf:hasDisjunctionsWith, omv: keyClasses, void:rootResource, mod: browsingUI, mod:sampleQueries, void:propertyPartition, void: classPartition, void: dataDump, void: openSearchDescription, void:uriLookupEndpoint, schema:comment, dct:created, dct:modified, dct:valid, dct: dateSubmitted, pav:curatedOn, omv:isOfType, mod:classesWithMoreThan25Children, mod:classesWithOneChild, mod:classesWithNoDefinition, mod:maxChildCount</td></tr></table>

<table><caption>## Annex 3: List of selected libraries and repositories taken into consideration for the evaluation of the F4 questions. The star symbol (*) distinguishes the ones supporting authentication mechanisms. The list is open for modification/suggestion on GitHub: https://github.com/akoportal/fairness/blob/master/src/main/resources/config/common/catalogs.config.json</caption><thead><tr><th></th><th>Name</th><th>URL</th></tr></thead><tbody><tr><td rowspan="10">Library</td><td>OBO Foundry</td><td>http://www.obofoundry.org/</td></tr><tr><td>CROP Ontology Curation Tool</td><td>http://www.cropontology.org/</td></tr><tr><td>Agrisemantics</td><td>http://vest.agrisemantics.org/</td></tr><tr><td>FAIRSharing</td><td>http://fairsharing.org/</td></tr><tr><td>ONKI Ontology Library Serv.</td><td>http://onki.fi/</td></tr><tr><td>MMI ORR</td><td>http://mmisw.org/</td></tr><tr><td>ROMULUS</td><td>http://thezfiles.co.za/ROMULUS/</td></tr><tr><td>DAML Ontology Library</td><td>http://www.daml.org/</td></tr><tr><td>Colore</td><td>http://stl.mie.utoronto.ca/colore/</td></tr><tr><td>BARTOC</td><td>http://bartoc.org/</td></tr></tbody></table>
**Annex 3:** List of selected libraries and repositories taken into consideration for the evaluation of the F4 questions. The star symbol (*) distinguishes the ones supporting authentication mechanisms. The list is open for modification/suggestion on GitHub: https://github.com/agroportal/fairness/blob/master/src/main/resources/config/common/catalogs.config.json (continued)

<table><thead><tr><th></th><th>Name</th><th>URL</th></tr></thead><tbody><tr><td rowspan="12">Repository</td><td>TaxoBank</td><td>http://www.taxobank.org/</td></tr><tr><td>TaxoBank</td><td>http://www.taxobank.org/</td></tr><tr><td>LusTRE</td><td>http://linkeddata.ge.imati.cnr.it/</td></tr><tr><td>LOV4IoT</td><td>http://lov4iot.appspot.com/</td></tr><tr><td>VOCAB OEG</td><td>http://vocab.linkeddata.es/</td></tr><tr><td>LiveSchema</td><td>http://liveschema.eu/</td></tr><tr><td>Protege Ontology library</td><td>http://protegewiki.stanford.edu/wiki/Protege_Ontology_Library</td></tr><tr><td>BioPortal *</td><td>http://bioportal.bioontology.org/</td></tr><tr><td>AgroPortal *</td><td>http://agroportal.lirmm.fr/</td></tr><tr><td>SIFR BioPortal *</td><td>http://bioportal.lirmm.fr/</td></tr><tr><td>MedPortal *</td><td>http://medportal.bmicc.cn/</td></tr><tr><td>LifeWatch EcoPortal *</td><td>http://ecoportal.lifewatchitaly.eu/</td></tr><tr><td>MatPortal *</td><td>http://matportal.org/</td></tr><tr><td>IndustryPortal</td><td>http://industryportal.enit.fr/</td></tr><tr><td>EBI Ontology Lookup Serv.</td><td>https://www.ebi.ac.uk/ols/</td></tr><tr><td>Ontobee</td><td>http://www.ontobee.org/</td></tr><tr><td>LOV</td><td>http://lov.linkeddata.es/dataset/lov</td></tr><tr><td>AberOwl</td><td>http://aber-owl.net/</td></tr><tr><td>OntoHub</td><td>http://ontohub.org/</td></tr><tr><td>CISMeF HeTOP*</td><td>http://www.hetop.eu/hetop/</td></tr><tr><td>FINTO</td><td>https://finto.fi/en/</td></tr><tr><td>ANDC RVA</td><td>https://vocabs.ardc.edu.au/</td></tr><tr><td>NERC NVS</td><td>https://vocab.nerc.ac.uk/</td></tr><tr><td>GFBIO IS</td><td>http://terminologies.gfbio.org/</td></tr><tr><td>LOTERRE</td><td>https://www.loterre.fr</td></tr><tr><td>Planteome</td><td>http://www.planteome.org/</td></tr><tr><td>CALIPER</td><td>http://datalab.review.fao.org/datalab/caliper/web/</td></tr></tbody></table>

**Annex 4:** O'FAIRe proposed classification of knowledge representation languages with evaluation (1 to 5 stars) of each characteristic required by sub-principle I1. Note: our interpretation of each characteristic is: Formal: How easy is it for machines to understand the semantics of the knowledge encoded in a resource? Accessible: How easy is it for machines to access and process the content of the resource? Shared: Is the representation language used widely adopted and shared? Is it supported by a well-known community? Broadly applicable: How much the language can be used for a wide range of ontologies and metadata representation independently of the task or the domain?

<table><thead><tr><th></th><th>Formal</th><th>Accessible</th><th>Shared</th><th>Broadly applicable</th><th>Total points</th></tr></thead><tbody><tr><td>OWL</td><td>*****</td><td>*****</td><td>***</td><td>*****</td><td>18</td></tr><tr><td>OBO</td><td>****</td><td>****</td><td>*</td><td>**</td><td>11</td></tr><tr><td>RDFS</td><td>**</td><td>*****</td><td>**</td><td>***</td><td>12</td></tr><tr><td>SKOS</td><td>***</td><td>*****</td><td>***</td><td>****</td><td>15</td></tr><tr><td>CSV</td><td>*</td><td>**</td><td>****</td><td>**</td><td>9</td></tr><tr><td>XML</td><td>*</td><td>***</td><td>****</td><td>**</td><td>10</td></tr><tr><td>PDF</td><td>-</td><td>-</td><td>****</td><td>*</td><td>5</td></tr><tr><td>TXT</td><td>-</td><td>*</td><td>*****</td><td>*</td><td>7</td></tr></tbody></table>
<table><caption>**Annex 5:** O'FAIRRe selected metadata vocabularies and their proposed evaluation with a coarse grain '4-level based FAIRness coefficient' manually assigned (from 1 to 4 stars); recognized by I2Q7. Available also on GitHub: https://github.com/agroportal/fairness/blob/master/src/main/resources/config/common/metadata.voc.config.json</caption><thead><tr><td>Prefix</td><td>Name</td><td>URL</td><td>Coef</td></tr></thead><tbody><tr><td>rdfs</td><td>RDF Schema</td><td>https://www.w3.org/TR/rdf-schema/</td><td>***</td></tr><tr><td>owl</td><td>OWL 2 Web Ontology Language</td><td>https://www.w3.org/TR/owl2-quick-reference/</td><td>***</td></tr><tr><td>skos</td><td>Simple Knowledge Organization System</td><td>https://www.w3.org/TR/skos-reference/</td><td>***</td></tr><tr><td>dc</td><td>Dublin Core (DCMI Metadata Terms)</td><td>http://purl.org/dc/elements/1.1/</td><td>***</td></tr><tr><td>dct terms</td><td>Dublin Core (DCMI Metadata Terms)</td><td>http://purl.org/dc/terms/</td><td>***</td></tr><tr><td>omv</td><td>Ontology Metadata Vocabulary</td><td>http://omv2.sourceforge.net/</td><td>**</td></tr><tr><td>mod</td><td>Metadata for Ontology Description and Publication</td><td>http://www.isibang.ac.in/ns/mod.html</td><td>**</td></tr><tr><td>door</td><td>Descriptive Ontology of Ontology Relations</td><td>http://oro.open.ac.uk/24326/1/keod09.pdf</td><td>*</td></tr><tr><td>voaf</td><td>Vocabulary of a Friend</td><td>http://purl.org/vocommons/voaf#</td><td>**</td></tr><tr><td>void</td><td>Vocabulary of Interlinked Data sets</td><td>https://www.w3.org/TR/void/</td><td>**</td></tr><tr><td>idot</td><td>identifiers.org Terms</td><td>http://identifiers.org/idot</td><td>*</td></tr><tr><td>vann</td><td>Vocabulary for annotating vocabulary descriptions</td><td>http://purl.org/vocab/vann/</td><td>**</td></tr><tr><td>dcat</td><td>Data Catalogue Vocabulary</td><td>https://www.w3.org/TR/vocab-dcat/</td><td>***</td></tr><tr><td>adms</td><td>Asset Description Metadata schema</td><td>https://www.w3.org/TR/vocab-adms/</td><td>**</td></tr><tr><td>schema</td><td>schema.org</td><td>http://schema.org/Dataset</td><td>***</td></tr><tr><td>foaf</td><td>Friend of a Friend Vocabulary</td><td>http://xmlns.com/foaf/spec/</td><td>***</td></tr><tr><td>doap</td><td>Description of a Project</td><td>https://github.com/edumbill/doap/wiki</td><td>**</td></tr><tr><td>cc</td><td>Creative Commons Rights Expression Language</td><td>http://creativecommons.org/ns</td><td>***</td></tr><tr><td>prov</td><td>Provenance Ontology</td><td>https://www.w3.org/TR/prov-o/</td><td>***</td></tr><tr><td>oboInowl</td><td>OboInOwl Mappings</td><td>http://www.geneontology.org/formats/oboInOwl#</td><td>**</td></tr><tr><td>sd</td><td>SPARQL 1.1 Service Description</td><td>https://www.w3.org/TR/sparql11-service-description/</td><td>*</td></tr><tr><td>nkos</td><td>Networked Knowledge Organization Systems Dublin Core Application Profile (NKOS AP)</td><td>http://nkos.slis.kent.edu/nkos-ap.html</td><td>*</td></tr><tr><td>cito</td><td>The citation Typing Ontology</td><td>http://purl.org/spar/cito/</td><td>*</td></tr><tr><td>wdrs</td><td>Protocol for Web Description Resources (POWDER): POWDER-S Vocabulary (WDRS)</td><td>https://www.w3.org/2007/05/powder-s</td><td>**</td></tr><tr><td>bibo</td><td>The Bibliographic Ontology</td><td>http://lov.okfn.org/dataset/lov/vocabs/bibo</td><td>**</td></tr><tr><td>disco</td><td>DDI-RDF Discovery Vocabulary</td><td>http://rdf-vocabulary.ddialliance.org/discovery.html</td><td>**</td></tr><tr><td>oa</td><td>Open Annotation Data Model</td><td>https://www.w3.org/ns/oa</td><td>***</td></tr><tr><td>iao</td><td>Information Artifact Ontology</td><td>http://purl.bioontology.org/ontology/IAO</td><td>**</td></tr><tr><td>sio</td><td>Semanticscience Integrated Ontology (SIO)</td><td>http://semanticscience.org/resource/</td><td>**</td></tr><tr><td>void-ext</td><td>Extensions to the Vocabulary of Interlinked Data sets (VoID)</td><td>http://ldf.fi/schema/void-ext/</td><td>*</td></tr><tr><td>ro</td><td>Relations Ontology</td><td></td><td>*</td></tr><tr><td>swoogle</td><td>Swoogle Ontology</td><td>http://daml.umbc.edu/ontologies/webofbelief/1.4/swoogle.owl#</td><td>**</td></tr><tr><td>edam</td><td>EDAM</td><td>http://purl.obolibrary.org/obo/edam#</td><td>**</td></tr></tbody></table>