# A Generic Framework to Better Understand and Compare FAIRness Measures

Philippe Lamarre, Jennie Andersen, Alban Gaignard, Sylvie Cazalens

► To cite this version:

Philippe Lamarre, Jennie Andersen, Alban Gaignard, Sylvie Cazalens. A Generic Framework to Better Understand and Compare FAIRness Measures. EKAW 2024, Nov 2024, Amsterdam, Netherlands. pp.291-308, 10.1007/978-3-031-77792-9_18. hal-04830524

HAL Id: hal-04830524

https://hal.science/hal-04830524v1

Submitted on 11 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# A generic framework to better understand and compare FAIRness measures

Philippe Lamarre, Jennie Andersen, Alban Gaignard, Sylvie Cazalens

EKAW 2024 - Amsterdam

Inserm

<figure><img src="image_2.png" /></figure>

Nantes
Université

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>
# FAIR principles ≠ technical specifications

* Largely **adopted** by research funders and organizations

* Key for more open & reproducible sciences

## However

* Principles, non-technical guidelines

* can be **interpreted** differently by specific communities

* **many** ways of **implementing** them

nature > scientific data > comment > article

Comment | Open access

Published: 15 March 2016

**The FAIR Guiding Principles for scientific data management and stewardship**

Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J.G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, ... Barend Mons

Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J.G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, ... Barend Mons

Goble, Jeffrey S. Grethe, ... Barend Mons + Show authors

Scientific Data 3, Article number: 160018 (2016) | Cite this article

766k Accesses | 5450 Citations | 2263 Altmetric | Metrics

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

EUROPEAN COMMISSION
Directorate-General for Research & Innovation

H2020 Programme

Guidelines on
FAIR Data Management in Horizon 2020
# Multiple FAIR assessments tools

<figure><img src="image_8.png" /></figure>

FAIR Data Self-Assessment Tool

Satisfyd

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

F-UJI

O'FAIRe

<figure><img src="image_11.png" /></figure>

* Check lists / questionnaires

* Automated tools

* Some tools are "community oriented"

* Some tools are "technologically opinionated"
# How FAIR is a bioinformatics software ?

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

https://bio.tools/seurat

* Why are results so different?

* Which tool should I use?

* What's behind the scene?

<figure><img src="image_14.png" /></figure>
One digital object ...

... but different FAIR results 😳
① We need a computable
framework for modeling
FAIR measures
Objectives

(i) **specify measures** in a uniform and computable **model**

(ii) propose specific **quantities** to **analyse and compare** FAIR assessment approaches
# Typical structure for FAIR assessment methods

* **Principles P:** the FAIR principles and their sub-principles

* **Indicators $I(\mathcal{M})$:** the specification of principles, (i.e. what has to be verified)

* **Implementations $Imp(\mathcal{M})$:** the implementations of the principles

<figure><img src="image_15.png" /></figure>

$$ \mathcal{M} = \underbrace{(V, E, \text{FAIR}, \diamond, w, v_{\text{max}}, D)}_{\text{structure}} \underbrace{\text{score}} $$
# Score computation

Max function: $v_{max}(i)$ returns the maximum reachable score for an indicator or an implementation

Weighting function: $w(n)$ returns the weight of the node $n$ compared to its siblings

Aggregation function : weighted sum or weighted average to collect scores

<figure><img src="image_16.png" /></figure>

$$ \mathcal{M} = \underbrace{(V, E, \text{FAIR}, \blackdiamond, w, v_{max}, D)}_{\text{structure}} \overline{\underline{\text{score}}} $$
Model instantiation with FAIR-Checker

<figure><img src="image_17.png" /></figure>

► The tool fits the tree-based structure {$P,I,Imp$}

► needs the maximum reachable score per implementation → 2

► needs an aggregation function → AVG (normalized scores)

► needs cummulative weights for each principle → $w \in [0,4]$
② We need quantities to characterize and compare FAIR measures
Quantifying the *granularity* of a measure

**Idea:** a global metric computed as the mean number of indicators per principle with at least one indicator. If we have one principle with many implementations → important granularity.

<figure><img src="image_18.png" /></figure>

F has 4 indicators, A has only 2 → principles have not the same granularity

$$ \text{granularity}(F) = 4/2 = 2; \text{ granularity}(\text{FAIR}) = 12/10 = 1.2 $$
Quantifying the coverage of a measure

**Idea:** a principle is considered as covered if it exists at least one implementation for the principle or, its sub-principles.

<figure><img src="image_19.png" /></figure>

F3 & F4 are never evaluated → F has 50% coverage
Quantifying the *impact* of principles

**Idea:** the impact of a principle corresponds to the global score obtained when all its underlying implementations are successful (without considering other principles).

<figure><img src="image_20.png" /></figure>

$$ \text{imp}(A) = 2*2 / 12*2 = 1/6 $$

$$ \text{imp}(F) = 4*2 / 12*2 = 2/6 $$

→ F contributes 2 times more compared to A in the global FAIR score
Experimental results
# Experimental setup

## Evaluated tools

automated: F-UJI, FAIR Evaluator, FOOPS!, O'FAIRe, FAIR-Checker,

questionnaires: ARDC, SATiFYD

## 10 selected web resources for FAIR assessments, covering

- datasets descriptions

- ontologies

- online courses

- bioinformatics software

- RDF files
# Do engines reach consensus on FAIR assessment ?

<table><thead><tr><th>Resource</th><th>F-UJI (%)</th><th>FAIR-Checker (%)</th><th>Std dev</th></tr></thead><tbody><tr><td>Dataset (PANGAEA) [31]</td><td>91</td><td>91.70</td><td>0.49</td></tr><tr><td>Gene Ontology (OLS) [21]</td><td>18</td><td>16.70</td><td>0.92</td></tr><tr><td>Dataset (Harvard Dataverse) [23]</td><td>75</td><td>79.20</td><td>2.97</td></tr><tr><td>Dataset (Kaggle) [26]</td><td>60</td><td>70.80</td><td>7.64</td></tr><tr><td>Online course (Moodle) [28]</td><td>4</td><td>16.70</td><td>8.98</td></tr><tr><td>Dataset (Governmental platform) [22]</td><td>52</td><td>70.80</td><td>13.29</td></tr><tr><td>Dataset (WHO) [39]</td><td>27</td><td>50.00</td><td>16.26</td></tr><tr><td>Training material (TeSS) [36]</td><td>39</td><td>70.80</td><td>22.49</td></tr><tr><td>Bioinformatics tool (bio.tools) [6]</td><td>18</td><td>54.20</td><td>25.60</td></tr><tr><td>Dataset (RDF metadata) [33]</td><td>43</td><td>87.50</td><td>31.47</td></tr></tbody></table>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

* Higher scores for FAIR-Checker

* Last two entries: std. dev. > 25 % ?
Coverage rates

<figure><img src="image_23.png" /></figure>

some principles (★) are not implemented → technical issues ? → interpretation ?

{O-FAIRe} → the only one with a full coverage (1)

Low coverage for questionnaires (no sub-principles)
# Granularities

<figure><img src="image_24.png" /></figure>

* O-FAIRe (targeting ontologies) has the most fine-grained evaluation (e.g. I2 and R1.2) → great diversity in metadata for ontologies

* FOOPS deeply investigate F1 (identifiers)
# Impacts

Are all principle
equally contributing
to the global FAIR
assesment score ?
→ "No ..."

<figure><img src="image_25.png" /></figure>

How to get a good FAIR
score with a minimal effort ?

* Pay attention to identifiers (F), license + provenance + domain-specific standards (R) if you use FOOPS!

* Not useful to spend energy on provenance or domain ontologies if you use FAIR-Evaluator ...

* ... but pay attention to it if you use F-UJI.
Conclusion

and future works
# Take-home message

* We need to understand the specifities of FAIR measurement tools

* We introduced a generic **model** for representing FAIR measures and computing their **granularity**, **coverage** and **impact**

* Some tools are "biased" (intentionally or not):

- they explore more deeply some dimensions

- which has an impact on the scores

* **Future works** include

- better investigating implementations (e.g. dependencies),
  with tools developers

- share these metadata on the web (e.g. DQV ontology)

- contribute to FAIR harmonization efforts with other communities
  (softwares, workflows, machine learning resources ...)
# Acknowledgements

<figure><img src="image_26.png" /><figcaption>Jennien Andersen</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>Philippe Lamare</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Sylvie Cazalens</figcaption></figure>

contact
alban.gaignard@univ-nantes.fr

Funding:

ANR AAPG2019 DeKaloG

French institute for Bioinformatics (Elixir-FR)

<figure><img src="image_29.png" /></figure>

RÉPUBLIQUE
FRANÇAISE

Liberté
Egalité
Fraternité

anr

<figure><img src="image_30.png" /></figure>

agence nationale
de la recherche

AU SERVICE DE LA SCIENCE

<figure><img src="image_31.png" /></figure>

IFB

INSTITUT FRANÇAIS DE BIOINFORMATIQUE

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

l'institut
du thorax